[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260810 18:50:03.479099 14467 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.14.32.254:37379
I20260810 18:50:03.481669 14467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:50:03.483557 14467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:03.520371 14477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:03.520435 14481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:03.522511 14467 server_base.cc:1034] running on GCE node
W20260810 18:50:03.520428 14479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:04.838814 14467 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:04.839172 14467 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:04.839351 14467 hybrid_clock.cc:648] HybridClock initialized: now 1786387804839330 us; error 0 us; skew 500 ppm
I20260810 18:50:04.840561 14467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:04.847462 14467 webserver.cc:460] Webserver started at http://127.14.32.254:36809/ using document root <none> and password file <none>
I20260810 18:50:04.848831 14467 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:04.849056 14467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:04.849684 14467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:04.857702 14467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786387803181742-14467-0/minicluster-data/master-0-root/instance:
uuid: "7fe0a77d39a24aba944a1d2f29ce4eda"
format_stamp: "Formatted at 2026-08-10 18:50:04 on dist-test-slave-n326"
I20260810 18:50:04.867983 14467 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.010s sys 0.001s
I20260810 18:50:04.874763 14494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:04.876662 14467 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260810 18:50:04.877094 14467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786387803181742-14467-0/minicluster-data/master-0-root
uuid: "7fe0a77d39a24aba944a1d2f29ce4eda"
format_stamp: "Formatted at 2026-08-10 18:50:04 on dist-test-slave-n326"
I20260810 18:50:04.877482 14467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786387803181742-14467-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786387803181742-14467-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786387803181742-14467-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:04.929049 14467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:04.930902 14467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:50:04.931430 14467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:05.016280 14467 rpc_server.cc:307] RPC server started. Bound to: 127.14.32.254:37379
I20260810 18:50:05.016377 14572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.32.254:37379 every 8 connection(s)
I20260810 18:50:05.027833 14573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:05.056520 14573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fe0a77d39a24aba944a1d2f29ce4eda: Bootstrap starting.
I20260810 18:50:05.064595 14573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fe0a77d39a24aba944a1d2f29ce4eda: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:05.067224 14573 log.cc:824] T 00000000000000000000000000000000 P 7fe0a77d39a24aba944a1d2f29ce4eda: Log is configured to *not* fsync() on all Append() calls
I20260810 18:50:05.072921 14573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fe0a77d39a24aba944a1d2f29ce4eda: No bootstrap required, opened a new log
I20260810 18:50:05.099082 14573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fe0a77d39a24aba944a1d2f29ce4eda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe0a77d39a24aba944a1d2f29ce4eda" member_type: VOTER }
I20260810 18:50:05.099773 14573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fe0a77d39a24aba944a1d2f29ce4eda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:05.100051 14573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fe0a77d39a24aba944a1d2f29ce4eda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fe0a77d39a24aba944a1d2f29ce4eda, State: Initialized, Role: FOLLOWER
I20260810 18:50:05.101095 14573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fe0a77d39a24aba944a1d2f29ce4eda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe0a77d39a24aba944a1d2f29ce4eda" member_type: VOTER }
I20260810 18:50:05.101632 14573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fe0a77d39a24aba944a1d2f29ce4eda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:50:05.101904 14573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fe0a77d39a24aba944a1d2f29ce4eda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:50:05.102185 14573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fe0a77d39a24aba944a1d2f29ce4eda [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:05.106220 14573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fe0a77d39a24aba944a1d2f29ce4eda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe0a77d39a24aba944a1d2f29ce4eda" member_type: VOTER }
I20260810 18:50:05.107209 14573 leader_election.cc:302] T 00000000000000000000000000000000 P 7fe0a77d39a24aba944a1d2f29ce4eda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fe0a77d39a24aba944a1d2f29ce4eda; no voters:
I20260810 18:50:05.109753 14573 leader_election.cc:288] T 00000000000000000000000000000000 P 7fe0a77d39a24aba944a1d2f29ce4eda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:50:05.110188 14578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fe0a77d39a24aba944a1d2f29ce4eda [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:05.112946 14578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fe0a77d39a24aba944a1d2f29ce4eda [term 1 LEADER]: Becoming Leader. State: Replica: 7fe0a77d39a24aba944a1d2f29ce4eda, State: Running, Role: LEADER
I20260810 18:50:05.114111 14578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fe0a77d39a24aba944a1d2f29ce4eda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe0a77d39a24aba944a1d2f29ce4eda" member_type: VOTER }
I20260810 18:50:05.115312 14573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fe0a77d39a24aba944a1d2f29ce4eda [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:50:05.131212 14579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fe0a77d39a24aba944a1d2f29ce4eda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fe0a77d39a24aba944a1d2f29ce4eda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe0a77d39a24aba944a1d2f29ce4eda" member_type: VOTER } }
I20260810 18:50:05.131131 14583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fe0a77d39a24aba944a1d2f29ce4eda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fe0a77d39a24aba944a1d2f29ce4eda. Latest consensus state: current_term: 1 leader_uuid: "7fe0a77d39a24aba944a1d2f29ce4eda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe0a77d39a24aba944a1d2f29ce4eda" member_type: VOTER } }
I20260810 18:50:05.131815 14579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fe0a77d39a24aba944a1d2f29ce4eda [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:05.131866 14583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fe0a77d39a24aba944a1d2f29ce4eda [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:05.135896 14603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:50:05.151527 14603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:50:05.153944 14467 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260810 18:50:05.173556 14603 catalog_manager.cc:1347] Generated new cluster ID: de5ed0f694a24fd184344fb83304107f
I20260810 18:50:05.173908 14603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:50:05.196142 14603 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:50:05.198254 14603 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:50:05.214103 14603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fe0a77d39a24aba944a1d2f29ce4eda: Generated new TSK 0
I20260810 18:50:05.215387 14603 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:50:05.220822 14467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:05.231171 14619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:05.233304 14623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:05.238193 14467 server_base.cc:1034] running on GCE node
W20260810 18:50:05.238583 14625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:05.239629 14467 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:05.239841 14467 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:05.240017 14467 hybrid_clock.cc:648] HybridClock initialized: now 1786387805240002 us; error 0 us; skew 500 ppm
I20260810 18:50:05.240602 14467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:05.245393 14467 webserver.cc:460] Webserver started at http://127.14.32.193:46547/ using document root <none> and password file <none>
I20260810 18:50:05.245944 14467 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:05.246157 14467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:05.246428 14467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:05.247777 14467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786387803181742-14467-0/minicluster-data/ts-0-root/instance:
uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
format_stamp: "Formatted at 2026-08-10 18:50:05 on dist-test-slave-n326"
I20260810 18:50:05.253923 14467 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 18:50:05.258641 14632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:05.259516 14467 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:50:05.259811 14467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786387803181742-14467-0/minicluster-data/ts-0-root
uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
format_stamp: "Formatted at 2026-08-10 18:50:05 on dist-test-slave-n326"
I20260810 18:50:05.260170 14467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786387803181742-14467-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786387803181742-14467-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786387803181742-14467-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:05.285940 14467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:05.287539 14467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:05.290808 14467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:05.296177 14467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:05.296459 14467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:05.296748 14467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:05.296901 14467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:05.406607 14467 rpc_server.cc:307] RPC server started. Bound to: 127.14.32.193:45253
I20260810 18:50:05.406816 14746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.32.193:45253 every 8 connection(s)
I20260810 18:50:05.414736 14467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:05.429016 14755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:05.429793 14756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:05.435170 14467 server_base.cc:1034] running on GCE node
W20260810 18:50:05.435249 14760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:05.436600 14467 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:05.436866 14467 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:05.437067 14467 hybrid_clock.cc:648] HybridClock initialized: now 1786387805437053 us; error 0 us; skew 500 ppm
I20260810 18:50:05.437848 14467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:05.442297 14467 webserver.cc:460] Webserver started at http://127.14.32.194:33267/ using document root <none> and password file <none>
I20260810 18:50:05.442998 14467 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:05.443274 14467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:05.443588 14467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:05.444962 14467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786387803181742-14467-0/minicluster-data/ts-1-root/instance:
uuid: "94af46e728ad452d91785ca419624f59"
format_stamp: "Formatted at 2026-08-10 18:50:05 on dist-test-slave-n326"
I20260810 18:50:05.448740 14748 heartbeater.cc:347] Connected to a master server at 127.14.32.254:37379
I20260810 18:50:05.449157 14748 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:05.450340 14748 heartbeater.cc:511] Master 127.14.32.254:37379 requested a full tablet report, sending...
I20260810 18:50:05.452219 14467 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260810 18:50:05.454253 14519 ts_manager.cc:194] Registered new tserver with Master: 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253)
I20260810 18:50:05.457546 14766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:05.458048 14519 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52334
I20260810 18:50:05.458498 14467 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260810 18:50:05.458828 14467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786387803181742-14467-0/minicluster-data/ts-1-root
uuid: "94af46e728ad452d91785ca419624f59"
format_stamp: "Formatted at 2026-08-10 18:50:05 on dist-test-slave-n326"
I20260810 18:50:05.459213 14467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786387803181742-14467-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786387803181742-14467-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786387803181742-14467-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:05.506897 14467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:05.508217 14467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:05.511768 14467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:05.515496 14467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:05.515821 14467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 18:50:05.516060 14467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:05.516194 14467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:05.636245 14467 rpc_server.cc:307] RPC server started. Bound to: 127.14.32.194:45285
I20260810 18:50:05.636385 14891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.32.194:45285 every 8 connection(s)
I20260810 18:50:05.648290 14467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:05.657618 14895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:05.662189 14897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:05.666319 14899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:05.666798 14892 heartbeater.cc:347] Connected to a master server at 127.14.32.254:37379
I20260810 18:50:05.667397 14892 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:05.668702 14892 heartbeater.cc:511] Master 127.14.32.254:37379 requested a full tablet report, sending...
I20260810 18:50:05.669519 14467 server_base.cc:1034] running on GCE node
I20260810 18:50:05.670684 14467 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:05.671005 14467 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:05.671223 14519 ts_manager.cc:194] Registered new tserver with Master: 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285)
I20260810 18:50:05.671316 14467 hybrid_clock.cc:648] HybridClock initialized: now 1786387805671297 us; error 0 us; skew 500 ppm
I20260810 18:50:05.672128 14467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:05.673602 14519 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52344
I20260810 18:50:05.675623 14467 webserver.cc:460] Webserver started at http://127.14.32.195:33955/ using document root <none> and password file <none>
I20260810 18:50:05.676172 14467 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:05.676393 14467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:05.676821 14467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:05.678323 14467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786387803181742-14467-0/minicluster-data/ts-2-root/instance:
uuid: "b847dd1039674d29a3c811bdfb6c7f51"
format_stamp: "Formatted at 2026-08-10 18:50:05 on dist-test-slave-n326"
I20260810 18:50:05.683552 14467 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260810 18:50:05.687153 14904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:05.688063 14467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 18:50:05.688383 14467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786387803181742-14467-0/minicluster-data/ts-2-root
uuid: "b847dd1039674d29a3c811bdfb6c7f51"
format_stamp: "Formatted at 2026-08-10 18:50:05 on dist-test-slave-n326"
I20260810 18:50:05.688673 14467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786387803181742-14467-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786387803181742-14467-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786387803181742-14467-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:05.716938 14467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:05.718238 14467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:05.719977 14467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:05.722155 14467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:05.722381 14467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:05.722616 14467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:05.722780 14467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:05.838682 14467 rpc_server.cc:307] RPC server started. Bound to: 127.14.32.195:42173
I20260810 18:50:05.838848 15017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.32.195:42173 every 8 connection(s)
I20260810 18:50:05.859591 15018 heartbeater.cc:347] Connected to a master server at 127.14.32.254:37379
I20260810 18:50:05.860096 15018 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:05.861056 15018 heartbeater.cc:511] Master 127.14.32.254:37379 requested a full tablet report, sending...
I20260810 18:50:05.863300 14519 ts_manager.cc:194] Registered new tserver with Master: b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:05.864245 14467 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.019953987s
I20260810 18:50:05.865187 14519 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52352
I20260810 18:50:05.869067 14467 test_util.cc:268] Using random seed: -761687054
I20260810 18:50:05.942368 14519 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52354:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:50:05.948939 14519 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:50:06.068925 14959 tablet_service.cc:1467] Processing CreateTablet for tablet 69e52bc7697d4473b77a1bd1f6df04f2 (DEFAULT_TABLE table=test-workload [id=4adc61ee6d4c4c4d946ef7ffd2145f24]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260810 18:50:06.074640 14955 tablet_service.cc:1467] Processing CreateTablet for tablet cd2a25bed1474a0692bcdfde37ab1d5b (DEFAULT_TABLE table=test-workload [id=4adc61ee6d4c4c4d946ef7ffd2145f24]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260810 18:50:06.075361 14957 tablet_service.cc:1467] Processing CreateTablet for tablet 7977ab6121414743ab57535b534ea4f9 (DEFAULT_TABLE table=test-workload [id=4adc61ee6d4c4c4d946ef7ffd2145f24]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260810 18:50:06.077623 14954 tablet_service.cc:1467] Processing CreateTablet for tablet 20ae3d07fdc14d4e9be86682e53a7d10 (DEFAULT_TABLE table=test-workload [id=4adc61ee6d4c4c4d946ef7ffd2145f24]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260810 18:50:06.079461 14953 tablet_service.cc:1467] Processing CreateTablet for tablet f14b3e767f224ce2b789dfcfd0414b89 (DEFAULT_TABLE table=test-workload [id=4adc61ee6d4c4c4d946ef7ffd2145f24]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260810 18:50:06.085108 14959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69e52bc7697d4473b77a1bd1f6df04f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.094405 14957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7977ab6121414743ab57535b534ea4f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.088660 14820 tablet_service.cc:1467] Processing CreateTablet for tablet 7977ab6121414743ab57535b534ea4f9 (DEFAULT_TABLE table=test-workload [id=4adc61ee6d4c4c4d946ef7ffd2145f24]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260810 18:50:06.094446 14822 tablet_service.cc:1467] Processing CreateTablet for tablet 69e52bc7697d4473b77a1bd1f6df04f2 (DEFAULT_TABLE table=test-workload [id=4adc61ee6d4c4c4d946ef7ffd2145f24]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260810 18:50:06.096529 14952 tablet_service.cc:1467] Processing CreateTablet for tablet a8187c40cb114a26b4da50c7bdbd4067 (DEFAULT_TABLE table=test-workload [id=4adc61ee6d4c4c4d946ef7ffd2145f24]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260810 18:50:06.103545 14953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f14b3e767f224ce2b789dfcfd0414b89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.098671 14951 tablet_service.cc:1467] Processing CreateTablet for tablet 4ef3949841d54dfcb19e1f25582bbeaa (DEFAULT_TABLE table=test-workload [id=4adc61ee6d4c4c4d946ef7ffd2145f24]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260810 18:50:06.107663 14820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7977ab6121414743ab57535b534ea4f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.104431 14811 tablet_service.cc:1467] Processing CreateTablet for tablet f14b3e767f224ce2b789dfcfd0414b89 (DEFAULT_TABLE table=test-workload [id=4adc61ee6d4c4c4d946ef7ffd2145f24]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260810 18:50:06.105640 14814 tablet_service.cc:1467] Processing CreateTablet for tablet 20ae3d07fdc14d4e9be86682e53a7d10 (DEFAULT_TABLE table=test-workload [id=4adc61ee6d4c4c4d946ef7ffd2145f24]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260810 18:50:06.100344 14947 tablet_service.cc:1467] Processing CreateTablet for tablet 9a2e5b7738d54ec99159df35d65b7b32 (DEFAULT_TABLE table=test-workload [id=4adc61ee6d4c4c4d946ef7ffd2145f24]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260810 18:50:06.105582 14818 tablet_service.cc:1467] Processing CreateTablet for tablet cd2a25bed1474a0692bcdfde37ab1d5b (DEFAULT_TABLE table=test-workload [id=4adc61ee6d4c4c4d946ef7ffd2145f24]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260810 18:50:06.109936 14949 tablet_service.cc:1467] Processing CreateTablet for tablet 395eacb81f614569895253d87e09b9ba (DEFAULT_TABLE table=test-workload [id=4adc61ee6d4c4c4d946ef7ffd2145f24]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260810 18:50:06.111774 14809 tablet_service.cc:1467] Processing CreateTablet for tablet a8187c40cb114a26b4da50c7bdbd4067 (DEFAULT_TABLE table=test-workload [id=4adc61ee6d4c4c4d946ef7ffd2145f24]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260810 18:50:06.114413 14952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8187c40cb114a26b4da50c7bdbd4067. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.119678 14818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd2a25bed1474a0692bcdfde37ab1d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.115736 14808 tablet_service.cc:1467] Processing CreateTablet for tablet 4ef3949841d54dfcb19e1f25582bbeaa (DEFAULT_TABLE table=test-workload [id=4adc61ee6d4c4c4d946ef7ffd2145f24]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260810 18:50:06.121076 14806 tablet_service.cc:1467] Processing CreateTablet for tablet 395eacb81f614569895253d87e09b9ba (DEFAULT_TABLE table=test-workload [id=4adc61ee6d4c4c4d946ef7ffd2145f24]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260810 18:50:06.120927 14805 tablet_service.cc:1467] Processing CreateTablet for tablet 9a2e5b7738d54ec99159df35d65b7b32 (DEFAULT_TABLE table=test-workload [id=4adc61ee6d4c4c4d946ef7ffd2145f24]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260810 18:50:06.125237 14811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f14b3e767f224ce2b789dfcfd0414b89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.126178 14808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ef3949841d54dfcb19e1f25582bbeaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.127961 14805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a2e5b7738d54ec99159df35d65b7b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.129276 14954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20ae3d07fdc14d4e9be86682e53a7d10. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 18:50:06.135706 14600 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260810 18:50:06.136332 14806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 395eacb81f614569895253d87e09b9ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.139425 14951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ef3949841d54dfcb19e1f25582bbeaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.147044 14955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd2a25bed1474a0692bcdfde37ab1d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.152560 14949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 395eacb81f614569895253d87e09b9ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.154031 14814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20ae3d07fdc14d4e9be86682e53a7d10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.156867 14809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8187c40cb114a26b4da50c7bdbd4067. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.158828 14822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69e52bc7697d4473b77a1bd1f6df04f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.167415 14947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a2e5b7738d54ec99159df35d65b7b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.228440 14684 tablet_service.cc:1467] Processing CreateTablet for tablet 69e52bc7697d4473b77a1bd1f6df04f2 (DEFAULT_TABLE table=test-workload [id=4adc61ee6d4c4c4d946ef7ffd2145f24]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260810 18:50:06.235018 14684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69e52bc7697d4473b77a1bd1f6df04f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.240180 14676 tablet_service.cc:1467] Processing CreateTablet for tablet 20ae3d07fdc14d4e9be86682e53a7d10 (DEFAULT_TABLE table=test-workload [id=4adc61ee6d4c4c4d946ef7ffd2145f24]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260810 18:50:06.248603 14676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20ae3d07fdc14d4e9be86682e53a7d10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.240409 14680 tablet_service.cc:1467] Processing CreateTablet for tablet 7977ab6121414743ab57535b534ea4f9 (DEFAULT_TABLE table=test-workload [id=4adc61ee6d4c4c4d946ef7ffd2145f24]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260810 18:50:06.243322 14679 tablet_service.cc:1467] Processing CreateTablet for tablet cd2a25bed1474a0692bcdfde37ab1d5b (DEFAULT_TABLE table=test-workload [id=4adc61ee6d4c4c4d946ef7ffd2145f24]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260810 18:50:06.253302 14675 tablet_service.cc:1467] Processing CreateTablet for tablet f14b3e767f224ce2b789dfcfd0414b89 (DEFAULT_TABLE table=test-workload [id=4adc61ee6d4c4c4d946ef7ffd2145f24]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260810 18:50:06.269002 14680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7977ab6121414743ab57535b534ea4f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.279435 14671 tablet_service.cc:1467] Processing CreateTablet for tablet 395eacb81f614569895253d87e09b9ba (DEFAULT_TABLE table=test-workload [id=4adc61ee6d4c4c4d946ef7ffd2145f24]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260810 18:50:06.284845 14670 tablet_service.cc:1467] Processing CreateTablet for tablet 9a2e5b7738d54ec99159df35d65b7b32 (DEFAULT_TABLE table=test-workload [id=4adc61ee6d4c4c4d946ef7ffd2145f24]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260810 18:50:06.287312 14679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd2a25bed1474a0692bcdfde37ab1d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.281829 14674 tablet_service.cc:1467] Processing CreateTablet for tablet a8187c40cb114a26b4da50c7bdbd4067 (DEFAULT_TABLE table=test-workload [id=4adc61ee6d4c4c4d946ef7ffd2145f24]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260810 18:50:06.283861 14673 tablet_service.cc:1467] Processing CreateTablet for tablet 4ef3949841d54dfcb19e1f25582bbeaa (DEFAULT_TABLE table=test-workload [id=4adc61ee6d4c4c4d946ef7ffd2145f24]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260810 18:50:06.296741 14670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a2e5b7738d54ec99159df35d65b7b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.308393 14673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ef3949841d54dfcb19e1f25582bbeaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.317323 14675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f14b3e767f224ce2b789dfcfd0414b89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.320834 14671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 395eacb81f614569895253d87e09b9ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.332882 14674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8187c40cb114a26b4da50c7bdbd4067. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.375921 15042 tablet_bootstrap.cc:492] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59: Bootstrap starting.
I20260810 18:50:06.392429 15045 tablet_bootstrap.cc:492] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51: Bootstrap starting.
I20260810 18:50:06.445303 15047 tablet_bootstrap.cc:492] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5: Bootstrap starting.
I20260810 18:50:06.444420 15042 tablet_bootstrap.cc:654] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:06.462433 14748 heartbeater.cc:503] Master 127.14.32.254:37379 was elected leader, sending a full tablet report...
I20260810 18:50:06.481666 15045 tablet_bootstrap.cc:654] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:06.486553 15047 tablet_bootstrap.cc:654] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:06.503299 15042 tablet_bootstrap.cc:492] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59: No bootstrap required, opened a new log
I20260810 18:50:06.504725 15042 ts_tablet_manager.cc:1397] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59: Time spent bootstrapping tablet: real 0.177s user 0.016s sys 0.013s
I20260810 18:50:06.508025 15045 tablet_bootstrap.cc:492] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51: No bootstrap required, opened a new log
I20260810 18:50:06.508774 15045 ts_tablet_manager.cc:1397] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51: Time spent bootstrapping tablet: real 0.125s user 0.010s sys 0.024s
I20260810 18:50:06.513962 15042 raft_consensus.cc:348] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:06.514436 15045 raft_consensus.cc:348] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:06.515111 15042 raft_consensus.cc:374] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:06.515226 15045 raft_consensus.cc:374] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:06.515647 15042 raft_consensus.cc:729] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94af46e728ad452d91785ca419624f59, State: Initialized, Role: FOLLOWER
I20260810 18:50:06.515761 15045 raft_consensus.cc:729] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b847dd1039674d29a3c811bdfb6c7f51, State: Initialized, Role: FOLLOWER
I20260810 18:50:06.516927 15042 consensus_queue.cc:260] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:06.516861 15045 consensus_queue.cc:260] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:06.521659 15047 tablet_bootstrap.cc:492] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5: No bootstrap required, opened a new log
I20260810 18:50:06.522187 15047 ts_tablet_manager.cc:1397] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5: Time spent bootstrapping tablet: real 0.089s user 0.016s sys 0.016s
I20260810 18:50:06.522716 14892 heartbeater.cc:503] Master 127.14.32.254:37379 was elected leader, sending a full tablet report...
I20260810 18:50:06.525767 15018 heartbeater.cc:503] Master 127.14.32.254:37379 was elected leader, sending a full tablet report...
I20260810 18:50:06.526360 15045 ts_tablet_manager.cc:1428] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51: Time spent starting tablet: real 0.017s user 0.007s sys 0.009s
I20260810 18:50:06.527881 15045 tablet_bootstrap.cc:492] T f14b3e767f224ce2b789dfcfd0414b89 P b847dd1039674d29a3c811bdfb6c7f51: Bootstrap starting.
I20260810 18:50:06.533860 15047 raft_consensus.cc:348] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:06.535254 15047 raft_consensus.cc:374] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:06.535851 15047 raft_consensus.cc:729] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49ef0f73074a4b6e9549f68ceb348ae5, State: Initialized, Role: FOLLOWER
I20260810 18:50:06.538512 15042 ts_tablet_manager.cc:1428] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59: Time spent starting tablet: real 0.033s user 0.008s sys 0.005s
I20260810 18:50:06.537763 15047 consensus_queue.cc:260] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:06.544415 15042 tablet_bootstrap.cc:492] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59: Bootstrap starting.
I20260810 18:50:06.553189 15047 ts_tablet_manager.cc:1428] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5: Time spent starting tablet: real 0.031s user 0.003s sys 0.015s
I20260810 18:50:06.554754 15047 tablet_bootstrap.cc:492] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5: Bootstrap starting.
I20260810 18:50:06.563303 15045 tablet_bootstrap.cc:654] T f14b3e767f224ce2b789dfcfd0414b89 P b847dd1039674d29a3c811bdfb6c7f51: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:06.575318 15047 tablet_bootstrap.cc:654] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:06.582502 15045 tablet_bootstrap.cc:492] T f14b3e767f224ce2b789dfcfd0414b89 P b847dd1039674d29a3c811bdfb6c7f51: No bootstrap required, opened a new log
I20260810 18:50:06.583161 15045 ts_tablet_manager.cc:1397] T f14b3e767f224ce2b789dfcfd0414b89 P b847dd1039674d29a3c811bdfb6c7f51: Time spent bootstrapping tablet: real 0.056s user 0.010s sys 0.020s
I20260810 18:50:06.586760 15042 tablet_bootstrap.cc:654] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:06.588282 15045 raft_consensus.cc:348] T f14b3e767f224ce2b789dfcfd0414b89 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:06.594805 15045 raft_consensus.cc:374] T f14b3e767f224ce2b789dfcfd0414b89 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:06.595358 15045 raft_consensus.cc:729] T f14b3e767f224ce2b789dfcfd0414b89 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b847dd1039674d29a3c811bdfb6c7f51, State: Initialized, Role: FOLLOWER
W20260810 18:50:06.596227 15019 tablet.cc:2369] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:50:06.596743 15045 consensus_queue.cc:260] T f14b3e767f224ce2b789dfcfd0414b89 P b847dd1039674d29a3c811bdfb6c7f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:06.601452 15045 ts_tablet_manager.cc:1428] T f14b3e767f224ce2b789dfcfd0414b89 P b847dd1039674d29a3c811bdfb6c7f51: Time spent starting tablet: real 0.018s user 0.010s sys 0.000s
I20260810 18:50:06.603052 15045 tablet_bootstrap.cc:492] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51: Bootstrap starting.
I20260810 18:50:06.612735 15047 tablet_bootstrap.cc:492] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5: No bootstrap required, opened a new log
I20260810 18:50:06.613615 15047 ts_tablet_manager.cc:1397] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5: Time spent bootstrapping tablet: real 0.059s user 0.023s sys 0.016s
I20260810 18:50:06.615808 15042 tablet_bootstrap.cc:492] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59: No bootstrap required, opened a new log
I20260810 18:50:06.616436 15042 ts_tablet_manager.cc:1397] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59: Time spent bootstrapping tablet: real 0.072s user 0.014s sys 0.019s
I20260810 18:50:06.619774 15047 raft_consensus.cc:348] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:06.624231 15047 raft_consensus.cc:374] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:06.624740 15047 raft_consensus.cc:729] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49ef0f73074a4b6e9549f68ceb348ae5, State: Initialized, Role: FOLLOWER
I20260810 18:50:06.626601 15045 tablet_bootstrap.cc:654] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:06.627434 15042 raft_consensus.cc:348] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:06.628444 15042 raft_consensus.cc:374] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:06.628870 15042 raft_consensus.cc:729] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94af46e728ad452d91785ca419624f59, State: Initialized, Role: FOLLOWER
I20260810 18:50:06.625945 15047 consensus_queue.cc:260] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:06.629879 15042 consensus_queue.cc:260] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:06.634383 15047 ts_tablet_manager.cc:1428] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5: Time spent starting tablet: real 0.020s user 0.008s sys 0.004s
I20260810 18:50:06.642591 15047 tablet_bootstrap.cc:492] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5: Bootstrap starting.
I20260810 18:50:06.653641 15042 ts_tablet_manager.cc:1428] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59: Time spent starting tablet: real 0.037s user 0.004s sys 0.008s
I20260810 18:50:06.655383 15047 tablet_bootstrap.cc:654] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:06.661177 15042 tablet_bootstrap.cc:492] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59: Bootstrap starting.
I20260810 18:50:06.666159 15055 raft_consensus.cc:482] T f14b3e767f224ce2b789dfcfd0414b89 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:06.666821 15055 raft_consensus.cc:504] T f14b3e767f224ce2b789dfcfd0414b89 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:06.671692 15055 leader_election.cc:288] T f14b3e767f224ce2b789dfcfd0414b89 P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285)
I20260810 18:50:06.694585 15047 tablet_bootstrap.cc:492] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5: No bootstrap required, opened a new log
I20260810 18:50:06.695487 15047 ts_tablet_manager.cc:1397] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5: Time spent bootstrapping tablet: real 0.053s user 0.007s sys 0.024s
I20260810 18:50:06.700843 15042 tablet_bootstrap.cc:654] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:06.705873 15055 raft_consensus.cc:482] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:06.706480 15055 raft_consensus.cc:504] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:06.708851 15055 leader_election.cc:288] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285)
I20260810 18:50:06.715003 15045 tablet_bootstrap.cc:492] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51: No bootstrap required, opened a new log
I20260810 18:50:06.715783 15045 ts_tablet_manager.cc:1397] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51: Time spent bootstrapping tablet: real 0.113s user 0.019s sys 0.052s
I20260810 18:50:06.714869 15047 raft_consensus.cc:348] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:06.716323 15047 raft_consensus.cc:374] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:06.716960 15047 raft_consensus.cc:729] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49ef0f73074a4b6e9549f68ceb348ae5, State: Initialized, Role: FOLLOWER
I20260810 18:50:06.718334 15047 consensus_queue.cc:260] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:06.721563 15045 raft_consensus.cc:348] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:06.722506 15045 raft_consensus.cc:374] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:06.722860 15045 raft_consensus.cc:729] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b847dd1039674d29a3c811bdfb6c7f51, State: Initialized, Role: FOLLOWER
I20260810 18:50:06.723234 15047 ts_tablet_manager.cc:1428] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5: Time spent starting tablet: real 0.025s user 0.003s sys 0.009s
I20260810 18:50:06.725584 15047 tablet_bootstrap.cc:492] T 9a2e5b7738d54ec99159df35d65b7b32 P 49ef0f73074a4b6e9549f68ceb348ae5: Bootstrap starting.
I20260810 18:50:06.723749 15045 consensus_queue.cc:260] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:06.733634 15057 raft_consensus.cc:482] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:06.736944 15057 raft_consensus.cc:504] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:06.748708 15045 ts_tablet_manager.cc:1428] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51: Time spent starting tablet: real 0.032s user 0.005s sys 0.006s
I20260810 18:50:06.749782 15057 leader_election.cc:288] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:06.752015 15045 tablet_bootstrap.cc:492] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51: Bootstrap starting.
I20260810 18:50:06.759756 15042 tablet_bootstrap.cc:492] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59: No bootstrap required, opened a new log
I20260810 18:50:06.761229 15042 ts_tablet_manager.cc:1397] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59: Time spent bootstrapping tablet: real 0.100s user 0.016s sys 0.039s
I20260810 18:50:06.768852 15055 raft_consensus.cc:482] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:06.768770 14704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20ae3d07fdc14d4e9be86682e53a7d10" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" is_pre_election: true
I20260810 18:50:06.769667 15055 raft_consensus.cc:504] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:06.771867 14706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14b3e767f224ce2b789dfcfd0414b89" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" is_pre_election: true
I20260810 18:50:06.770437 14704 raft_consensus.cc:2454] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b847dd1039674d29a3c811bdfb6c7f51 in term 0.
I20260810 18:50:06.779394 15047 tablet_bootstrap.cc:654] T 9a2e5b7738d54ec99159df35d65b7b32 P 49ef0f73074a4b6e9549f68ceb348ae5: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:06.780296 15055 leader_election.cc:288] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285)
I20260810 18:50:06.785140 14701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395eacb81f614569895253d87e09b9ba" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" is_pre_election: true
W20260810 18:50:06.787461 14911 leader_election.cc:341] T f14b3e767f224ce2b789dfcfd0414b89 P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:06.788524 14701 raft_consensus.cc:2454] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b847dd1039674d29a3c811bdfb6c7f51 in term 0.
I20260810 18:50:06.790387 14911 leader_election.cc:302] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5, b847dd1039674d29a3c811bdfb6c7f51; no voters:
I20260810 18:50:06.793865 15055 raft_consensus.cc:2781] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:06.794456 15055 raft_consensus.cc:482] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:06.794255 14911 leader_election.cc:302] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5, b847dd1039674d29a3c811bdfb6c7f51; no voters:
I20260810 18:50:06.795449 15055 raft_consensus.cc:3037] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:06.800226 15042 raft_consensus.cc:348] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:06.807830 15042 raft_consensus.cc:374] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:06.809876 15042 raft_consensus.cc:729] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94af46e728ad452d91785ca419624f59, State: Initialized, Role: FOLLOWER
I20260810 18:50:06.812202 15053 raft_consensus.cc:482] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:06.813912 15042 consensus_queue.cc:260] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:06.815052 15053 raft_consensus.cc:504] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:06.835312 15053 leader_election.cc:288] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:06.864902 15055 raft_consensus.cc:504] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
W20260810 18:50:06.879987 14911 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:45253, user_credentials={real_user=slave}} blocked reactor thread for 85877.5us
I20260810 18:50:06.886415 14843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395eacb81f614569895253d87e09b9ba" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94af46e728ad452d91785ca419624f59" is_pre_election: true
I20260810 18:50:06.890772 15047 tablet_bootstrap.cc:492] T 9a2e5b7738d54ec99159df35d65b7b32 P 49ef0f73074a4b6e9549f68ceb348ae5: No bootstrap required, opened a new log
I20260810 18:50:06.892848 15047 ts_tablet_manager.cc:1397] T 9a2e5b7738d54ec99159df35d65b7b32 P 49ef0f73074a4b6e9549f68ceb348ae5: Time spent bootstrapping tablet: real 0.168s user 0.023s sys 0.052s
I20260810 18:50:06.905606 15055 leader_election.cc:288] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 election: Requested vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285)
W20260810 18:50:06.906577 14642 leader_election.cc:341] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:06.909201 15055 raft_consensus.cc:2781] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:06.910560 15055 raft_consensus.cc:482] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:06.912322 15055 raft_consensus.cc:3037] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:06.928223 15045 tablet_bootstrap.cc:654] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:06.929589 15042 ts_tablet_manager.cc:1428] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59: Time spent starting tablet: real 0.167s user 0.008s sys 0.035s
I20260810 18:50:06.932940 14701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20ae3d07fdc14d4e9be86682e53a7d10" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
I20260810 18:50:06.936203 14701 raft_consensus.cc:3037] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:06.933838 15047 raft_consensus.cc:348] T 9a2e5b7738d54ec99159df35d65b7b32 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:06.935983 14843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14b3e767f224ce2b789dfcfd0414b89" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94af46e728ad452d91785ca419624f59" is_pre_election: true
I20260810 18:50:06.936374 15042 tablet_bootstrap.cc:492] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59: Bootstrap starting.
I20260810 18:50:06.938616 15047 raft_consensus.cc:374] T 9a2e5b7738d54ec99159df35d65b7b32 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:06.942114 15047 raft_consensus.cc:729] T 9a2e5b7738d54ec99159df35d65b7b32 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49ef0f73074a4b6e9549f68ceb348ae5, State: Initialized, Role: FOLLOWER
W20260810 18:50:06.945231 14911 leader_election.cc:341] T f14b3e767f224ce2b789dfcfd0414b89 P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:06.946202 14843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20ae3d07fdc14d4e9be86682e53a7d10" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94af46e728ad452d91785ca419624f59"
I20260810 18:50:06.947501 14843 raft_consensus.cc:3037] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:06.941613 14841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20ae3d07fdc14d4e9be86682e53a7d10" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94af46e728ad452d91785ca419624f59" is_pre_election: true
I20260810 18:50:06.947250 14911 leader_election.cc:302] T f14b3e767f224ce2b789dfcfd0414b89 P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b847dd1039674d29a3c811bdfb6c7f51; no voters: 49ef0f73074a4b6e9549f68ceb348ae5, 94af46e728ad452d91785ca419624f59
I20260810 18:50:06.950554 14840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395eacb81f614569895253d87e09b9ba" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94af46e728ad452d91785ca419624f59" is_pre_election: true
I20260810 18:50:06.948112 14982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395eacb81f614569895253d87e09b9ba" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51" is_pre_election: true
I20260810 18:50:06.953256 15047 consensus_queue.cc:260] T 9a2e5b7738d54ec99159df35d65b7b32 P 49ef0f73074a4b6e9549f68ceb348ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:06.959913 15079 raft_consensus.cc:2726] T f14b3e767f224ce2b789dfcfd0414b89 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 18:50:06.963492 14911 leader_election.cc:341] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:06.970333 15055 raft_consensus.cc:504] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:06.973954 15047 ts_tablet_manager.cc:1428] T 9a2e5b7738d54ec99159df35d65b7b32 P 49ef0f73074a4b6e9549f68ceb348ae5: Time spent starting tablet: real 0.081s user 0.011s sys 0.007s
I20260810 18:50:06.976696 14982 raft_consensus.cc:2379] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49ef0f73074a4b6e9549f68ceb348ae5 in current term 1: Already voted for candidate b847dd1039674d29a3c811bdfb6c7f51 in this term.
I20260810 18:50:06.977842 15047 tablet_bootstrap.cc:492] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5: Bootstrap starting.
I20260810 18:50:06.986320 15055 leader_election.cc:288] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 election: Requested vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285)
I20260810 18:50:06.989859 14704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395eacb81f614569895253d87e09b9ba" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
I20260810 18:50:06.990823 14704 raft_consensus.cc:3037] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:06.990906 14843 raft_consensus.cc:2454] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b847dd1039674d29a3c811bdfb6c7f51 in term 1.
I20260810 18:50:06.995151 14911 leader_election.cc:302] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94af46e728ad452d91785ca419624f59, b847dd1039674d29a3c811bdfb6c7f51; no voters:
I20260810 18:50:06.996831 14843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395eacb81f614569895253d87e09b9ba" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94af46e728ad452d91785ca419624f59"
I20260810 18:50:06.998926 15055 raft_consensus.cc:2781] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:07.002535 15055 raft_consensus.cc:686] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 LEADER]: Becoming Leader. State: Replica: b847dd1039674d29a3c811bdfb6c7f51, State: Running, Role: LEADER
W20260810 18:50:07.005580 14911 leader_election.cc:341] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:07.012521 15082 raft_consensus.cc:482] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:07.013530 15082 raft_consensus.cc:504] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.029604 15082 leader_election.cc:288] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:07.032244 15042 tablet_bootstrap.cc:654] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:07.038445 14704 raft_consensus.cc:2454] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b847dd1039674d29a3c811bdfb6c7f51 in term 1.
I20260810 18:50:07.002126 14701 raft_consensus.cc:2454] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b847dd1039674d29a3c811bdfb6c7f51 in term 1.
I20260810 18:50:07.044445 14981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2a25bed1474a0692bcdfde37ab1d5b" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51" is_pre_election: true
I20260810 18:50:07.045337 15055 consensus_queue.cc:237] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.046236 14979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef3949841d54dfcb19e1f25582bbeaa" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51" is_pre_election: true
W20260810 18:50:07.049795 14777 leader_election.cc:341] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173): Illegal state: must be running to vote when last-logged opid is not known
W20260810 18:50:07.051522 14777 leader_election.cc:341] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:07.051713 14911 leader_election.cc:302] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5, b847dd1039674d29a3c811bdfb6c7f51; no voters: 94af46e728ad452d91785ca419624f59
I20260810 18:50:07.055557 15079 raft_consensus.cc:2781] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:07.057068 15057 raft_consensus.cc:482] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:07.058323 14701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef3949841d54dfcb19e1f25582bbeaa" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" is_pre_election: true
I20260810 18:50:07.057973 15057 raft_consensus.cc:504] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.057135 14704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2a25bed1474a0692bcdfde37ab1d5b" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" is_pre_election: true
W20260810 18:50:07.065768 14777 leader_election.cc:341] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:07.066712 14777 leader_election.cc:302] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94af46e728ad452d91785ca419624f59; no voters: 49ef0f73074a4b6e9549f68ceb348ae5, b847dd1039674d29a3c811bdfb6c7f51
I20260810 18:50:07.067948 15082 raft_consensus.cc:2726] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:50:07.068635 15045 tablet_bootstrap.cc:492] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51: No bootstrap required, opened a new log
I20260810 18:50:07.071326 14843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7977ab6121414743ab57535b534ea4f9" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94af46e728ad452d91785ca419624f59" is_pre_election: true
I20260810 18:50:07.071676 15047 tablet_bootstrap.cc:654] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:07.068949 15057 leader_election.cc:288] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:07.072034 14979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7977ab6121414743ab57535b534ea4f9" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51" is_pre_election: true
I20260810 18:50:07.073509 15045 ts_tablet_manager.cc:1397] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51: Time spent bootstrapping tablet: real 0.322s user 0.020s sys 0.031s
W20260810 18:50:07.075822 14642 leader_election.cc:341] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:07.080026 15042 tablet_bootstrap.cc:492] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59: No bootstrap required, opened a new log
I20260810 18:50:07.080580 15042 ts_tablet_manager.cc:1397] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59: Time spent bootstrapping tablet: real 0.146s user 0.014s sys 0.017s
W20260810 18:50:07.084270 14642 leader_election.cc:341] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:07.074849 15079 raft_consensus.cc:686] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 1 LEADER]: Becoming Leader. State: Replica: b847dd1039674d29a3c811bdfb6c7f51, State: Running, Role: LEADER
I20260810 18:50:07.086351 14642 leader_election.cc:302] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5; no voters: 94af46e728ad452d91785ca419624f59, b847dd1039674d29a3c811bdfb6c7f51
I20260810 18:50:07.099869 15057 raft_consensus.cc:2726] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:50:07.100262 14642 leader_election.cc:302] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5; no voters: 94af46e728ad452d91785ca419624f59, b847dd1039674d29a3c811bdfb6c7f51
I20260810 18:50:07.102095 15057 raft_consensus.cc:2726] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:50:07.103595 15079 consensus_queue.cc:237] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.105177 15057 raft_consensus.cc:482] T 9a2e5b7738d54ec99159df35d65b7b32 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:07.105968 15057 raft_consensus.cc:504] T 9a2e5b7738d54ec99159df35d65b7b32 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
W20260810 18:50:07.075062 14777 leader_election.cc:341] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:07.081753 15045 raft_consensus.cc:348] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.115586 15045 raft_consensus.cc:374] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:07.116101 15045 raft_consensus.cc:729] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b847dd1039674d29a3c811bdfb6c7f51, State: Initialized, Role: FOLLOWER
I20260810 18:50:07.115597 14777 leader_election.cc:302] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94af46e728ad452d91785ca419624f59; no voters: 49ef0f73074a4b6e9549f68ceb348ae5, b847dd1039674d29a3c811bdfb6c7f51
I20260810 18:50:07.117273 15045 consensus_queue.cc:260] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.125342 15055 raft_consensus.cc:482] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:07.126304 15055 raft_consensus.cc:504] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
W20260810 18:50:07.126567 14777 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:45253, user_credentials={real_user=slave}} blocked reactor thread for 51832.9us
I20260810 18:50:07.126974 15045 ts_tablet_manager.cc:1428] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51: Time spent starting tablet: real 0.051s user 0.009s sys 0.001s
I20260810 18:50:07.118652 15082 raft_consensus.cc:2726] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:50:07.128472 15045 tablet_bootstrap.cc:492] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51: Bootstrap starting.
I20260810 18:50:07.131398 15057 leader_election.cc:288] T 9a2e5b7738d54ec99159df35d65b7b32 P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:07.133509 15055 leader_election.cc:288] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285)
I20260810 18:50:07.135435 14843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a2e5b7738d54ec99159df35d65b7b32" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94af46e728ad452d91785ca419624f59" is_pre_election: true
W20260810 18:50:07.138439 14600 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260810 18:50:07.138257 15042 raft_consensus.cc:348] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.138861 14982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a2e5b7738d54ec99159df35d65b7b32" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51" is_pre_election: true
I20260810 18:50:07.139533 15042 raft_consensus.cc:374] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:07.139627 14704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7977ab6121414743ab57535b534ea4f9" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" is_pre_election: true
I20260810 18:50:07.140049 15042 raft_consensus.cc:729] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94af46e728ad452d91785ca419624f59, State: Initialized, Role: FOLLOWER
I20260810 18:50:07.140700 14704 raft_consensus.cc:2454] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b847dd1039674d29a3c811bdfb6c7f51 in term 0.
I20260810 18:50:07.141178 15042 consensus_queue.cc:260] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.143359 14911 leader_election.cc:302] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5, b847dd1039674d29a3c811bdfb6c7f51; no voters:
I20260810 18:50:07.143757 14840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7977ab6121414743ab57535b534ea4f9" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94af46e728ad452d91785ca419624f59" is_pre_election: true
I20260810 18:50:07.144511 15055 raft_consensus.cc:2781] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:07.145103 15055 raft_consensus.cc:482] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:07.145658 15055 raft_consensus.cc:3037] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Advancing to term 1
W20260810 18:50:07.141248 14642 leader_election.cc:341] T 9a2e5b7738d54ec99159df35d65b7b32 P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173): Illegal state: must be running to vote when last-logged opid is not known
W20260810 18:50:07.147269 14642 leader_election.cc:341] T 9a2e5b7738d54ec99159df35d65b7b32 P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:07.147749 14642 leader_election.cc:302] T 9a2e5b7738d54ec99159df35d65b7b32 P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5; no voters: 94af46e728ad452d91785ca419624f59, b847dd1039674d29a3c811bdfb6c7f51
W20260810 18:50:07.149518 14911 leader_election.cc:341] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:07.149192 15057 raft_consensus.cc:2726] T 9a2e5b7738d54ec99159df35d65b7b32 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:50:07.180842 15055 raft_consensus.cc:504] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.185789 14704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7977ab6121414743ab57535b534ea4f9" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
I20260810 18:50:07.186707 14840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7977ab6121414743ab57535b534ea4f9" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94af46e728ad452d91785ca419624f59"
W20260810 18:50:07.189018 14911 leader_election.cc:341] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:07.186749 14704 raft_consensus.cc:3037] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:07.207943 15047 tablet_bootstrap.cc:492] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5: No bootstrap required, opened a new log
I20260810 18:50:07.208465 15047 ts_tablet_manager.cc:1397] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5: Time spent bootstrapping tablet: real 0.231s user 0.019s sys 0.060s
I20260810 18:50:07.209175 15055 leader_election.cc:288] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 election: Requested vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285)
I20260810 18:50:07.206235 14518 catalog_manager.cc:5495] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 reported cstate change: term changed from 0 to 1, leader changed from <none> to b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195). New cstate: current_term: 1 leader_uuid: "b847dd1039674d29a3c811bdfb6c7f51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } health_report { overall_health: HEALTHY } } }
I20260810 18:50:07.212102 15047 raft_consensus.cc:348] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.213006 15042 ts_tablet_manager.cc:1428] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59: Time spent starting tablet: real 0.132s user 0.007s sys 0.011s
I20260810 18:50:07.216675 15042 tablet_bootstrap.cc:492] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59: Bootstrap starting.
I20260810 18:50:07.213490 14704 raft_consensus.cc:2454] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b847dd1039674d29a3c811bdfb6c7f51 in term 1.
I20260810 18:50:07.221369 14911 leader_election.cc:302] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5, b847dd1039674d29a3c811bdfb6c7f51; no voters: 94af46e728ad452d91785ca419624f59
I20260810 18:50:07.223287 15055 raft_consensus.cc:2781] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:07.224351 15055 raft_consensus.cc:686] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 LEADER]: Becoming Leader. State: Replica: b847dd1039674d29a3c811bdfb6c7f51, State: Running, Role: LEADER
I20260810 18:50:07.215428 15047 raft_consensus.cc:374] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:07.217005 14518 catalog_manager.cc:5495] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 reported cstate change: term changed from 0 to 1, leader changed from <none> to b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195). New cstate: current_term: 1 leader_uuid: "b847dd1039674d29a3c811bdfb6c7f51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } health_report { overall_health: HEALTHY } } }
I20260810 18:50:07.226823 15055 consensus_queue.cc:237] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.225716 15045 tablet_bootstrap.cc:654] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:07.226176 15047 raft_consensus.cc:729] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49ef0f73074a4b6e9549f68ceb348ae5, State: Initialized, Role: FOLLOWER
I20260810 18:50:07.234598 15047 consensus_queue.cc:260] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.243046 15042 tablet_bootstrap.cc:654] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:07.243822 15047 ts_tablet_manager.cc:1428] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5: Time spent starting tablet: real 0.035s user 0.011s sys 0.000s
I20260810 18:50:07.244799 15047 tablet_bootstrap.cc:492] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5: Bootstrap starting.
I20260810 18:50:07.255393 15047 tablet_bootstrap.cc:654] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:07.265113 15042 tablet_bootstrap.cc:492] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59: No bootstrap required, opened a new log
I20260810 18:50:07.265858 15042 ts_tablet_manager.cc:1397] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59: Time spent bootstrapping tablet: real 0.050s user 0.016s sys 0.011s
I20260810 18:50:07.267463 15045 tablet_bootstrap.cc:492] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51: No bootstrap required, opened a new log
I20260810 18:50:07.268126 15045 ts_tablet_manager.cc:1397] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51: Time spent bootstrapping tablet: real 0.140s user 0.013s sys 0.015s
I20260810 18:50:07.271953 15042 raft_consensus.cc:348] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } }
I20260810 18:50:07.272923 15042 raft_consensus.cc:374] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:07.273432 15042 raft_consensus.cc:729] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94af46e728ad452d91785ca419624f59, State: Initialized, Role: FOLLOWER
I20260810 18:50:07.274566 15042 consensus_queue.cc:260] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } }
I20260810 18:50:07.276074 15045 raft_consensus.cc:348] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.277187 15045 raft_consensus.cc:374] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:07.279600 15045 raft_consensus.cc:729] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b847dd1039674d29a3c811bdfb6c7f51, State: Initialized, Role: FOLLOWER
I20260810 18:50:07.281308 15045 consensus_queue.cc:260] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.282495 15042 ts_tablet_manager.cc:1428] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59: Time spent starting tablet: real 0.016s user 0.006s sys 0.006s
I20260810 18:50:07.283815 15042 tablet_bootstrap.cc:492] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59: Bootstrap starting.
I20260810 18:50:07.291647 15045 ts_tablet_manager.cc:1428] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51: Time spent starting tablet: real 0.021s user 0.000s sys 0.013s
I20260810 18:50:07.293273 15045 tablet_bootstrap.cc:492] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51: Bootstrap starting.
I20260810 18:50:07.313143 15047 tablet_bootstrap.cc:492] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5: No bootstrap required, opened a new log
I20260810 18:50:07.313520 15045 tablet_bootstrap.cc:654] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:07.313951 15047 ts_tablet_manager.cc:1397] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5: Time spent bootstrapping tablet: real 0.069s user 0.012s sys 0.027s
I20260810 18:50:07.316613 15042 tablet_bootstrap.cc:654] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:07.320315 15047 raft_consensus.cc:348] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.321484 15047 raft_consensus.cc:374] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:07.321913 15047 raft_consensus.cc:729] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49ef0f73074a4b6e9549f68ceb348ae5, State: Initialized, Role: FOLLOWER
I20260810 18:50:07.323523 15047 consensus_queue.cc:260] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.329609 15047 ts_tablet_manager.cc:1428] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260810 18:50:07.331314 15047 tablet_bootstrap.cc:492] T 4ef3949841d54dfcb19e1f25582bbeaa P 49ef0f73074a4b6e9549f68ceb348ae5: Bootstrap starting.
I20260810 18:50:07.348801 15042 tablet_bootstrap.cc:492] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59: No bootstrap required, opened a new log
I20260810 18:50:07.349663 15042 ts_tablet_manager.cc:1397] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59: Time spent bootstrapping tablet: real 0.066s user 0.023s sys 0.016s
I20260810 18:50:07.363085 15047 tablet_bootstrap.cc:654] T 4ef3949841d54dfcb19e1f25582bbeaa P 49ef0f73074a4b6e9549f68ceb348ae5: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:07.360235 14519 catalog_manager.cc:5495] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 reported cstate change: term changed from 0 to 1, leader changed from <none> to b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195). New cstate: current_term: 1 leader_uuid: "b847dd1039674d29a3c811bdfb6c7f51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } health_report { overall_health: HEALTHY } } }
I20260810 18:50:07.371471 15042 raft_consensus.cc:348] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.375497 15042 raft_consensus.cc:374] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:07.376183 15042 raft_consensus.cc:729] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94af46e728ad452d91785ca419624f59, State: Initialized, Role: FOLLOWER
I20260810 18:50:07.377123 15042 consensus_queue.cc:260] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.378607 15045 tablet_bootstrap.cc:492] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51: No bootstrap required, opened a new log
I20260810 18:50:07.379244 15045 ts_tablet_manager.cc:1397] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51: Time spent bootstrapping tablet: real 0.086s user 0.022s sys 0.031s
I20260810 18:50:07.387326 15045 raft_consensus.cc:348] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.388721 15045 raft_consensus.cc:374] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:07.389586 15045 raft_consensus.cc:729] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b847dd1039674d29a3c811bdfb6c7f51, State: Initialized, Role: FOLLOWER
I20260810 18:50:07.391681 15042 ts_tablet_manager.cc:1428] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59: Time spent starting tablet: real 0.042s user 0.004s sys 0.012s
I20260810 18:50:07.391288 15045 consensus_queue.cc:260] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.396839 15045 ts_tablet_manager.cc:1428] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51: Time spent starting tablet: real 0.017s user 0.009s sys 0.003s
I20260810 18:50:07.400498 15042 tablet_bootstrap.cc:492] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59: Bootstrap starting.
I20260810 18:50:07.410457 15045 tablet_bootstrap.cc:492] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51: Bootstrap starting.
I20260810 18:50:07.413775 15047 tablet_bootstrap.cc:492] T 4ef3949841d54dfcb19e1f25582bbeaa P 49ef0f73074a4b6e9549f68ceb348ae5: No bootstrap required, opened a new log
I20260810 18:50:07.414559 15047 ts_tablet_manager.cc:1397] T 4ef3949841d54dfcb19e1f25582bbeaa P 49ef0f73074a4b6e9549f68ceb348ae5: Time spent bootstrapping tablet: real 0.084s user 0.030s sys 0.019s
I20260810 18:50:07.423211 15047 raft_consensus.cc:348] T 4ef3949841d54dfcb19e1f25582bbeaa P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.424521 15047 raft_consensus.cc:374] T 4ef3949841d54dfcb19e1f25582bbeaa P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:07.424458 15082 raft_consensus.cc:482] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:07.425285 15082 raft_consensus.cc:504] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.427975 15045 tablet_bootstrap.cc:654] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:07.428175 15042 tablet_bootstrap.cc:654] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:07.430043 14704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14b3e767f224ce2b789dfcfd0414b89" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" is_pre_election: true
W20260810 18:50:07.432487 14777 leader_election.cc:341] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:07.433636 15082 leader_election.cc:288] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:07.436240 14982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14b3e767f224ce2b789dfcfd0414b89" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51" is_pre_election: true
I20260810 18:50:07.437211 14982 raft_consensus.cc:2454] T f14b3e767f224ce2b789dfcfd0414b89 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94af46e728ad452d91785ca419624f59 in term 0.
I20260810 18:50:07.442976 14777 leader_election.cc:302] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 94af46e728ad452d91785ca419624f59, b847dd1039674d29a3c811bdfb6c7f51; no voters: 49ef0f73074a4b6e9549f68ceb348ae5
I20260810 18:50:07.444559 15082 raft_consensus.cc:2781] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:07.445063 15082 raft_consensus.cc:482] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:07.445509 15082 raft_consensus.cc:3037] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:07.451180 15047 raft_consensus.cc:729] T 4ef3949841d54dfcb19e1f25582bbeaa P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49ef0f73074a4b6e9549f68ceb348ae5, State: Initialized, Role: FOLLOWER
I20260810 18:50:07.452402 15047 consensus_queue.cc:260] T 4ef3949841d54dfcb19e1f25582bbeaa P 49ef0f73074a4b6e9549f68ceb348ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.455097 15047 ts_tablet_manager.cc:1428] T 4ef3949841d54dfcb19e1f25582bbeaa P 49ef0f73074a4b6e9549f68ceb348ae5: Time spent starting tablet: real 0.040s user 0.001s sys 0.010s
I20260810 18:50:07.461166 15047 tablet_bootstrap.cc:492] T f14b3e767f224ce2b789dfcfd0414b89 P 49ef0f73074a4b6e9549f68ceb348ae5: Bootstrap starting.
I20260810 18:50:07.467221 15082 raft_consensus.cc:504] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.476022 15055 consensus_queue.cc:1035] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:50:07.476747 14704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14b3e767f224ce2b789dfcfd0414b89" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
I20260810 18:50:07.479211 15042 tablet_bootstrap.cc:492] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59: No bootstrap required, opened a new log
W20260810 18:50:07.478886 14777 leader_election.cc:341] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:07.485262 15042 ts_tablet_manager.cc:1397] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59: Time spent bootstrapping tablet: real 0.085s user 0.032s sys 0.025s
I20260810 18:50:07.487437 15082 leader_election.cc:288] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 1 election: Requested vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:07.489065 14982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14b3e767f224ce2b789dfcfd0414b89" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
I20260810 18:50:07.490053 14982 raft_consensus.cc:3037] T f14b3e767f224ce2b789dfcfd0414b89 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:07.492949 15045 tablet_bootstrap.cc:492] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51: No bootstrap required, opened a new log
I20260810 18:50:07.496716 15042 raft_consensus.cc:348] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.497943 15042 raft_consensus.cc:374] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:07.498488 15042 raft_consensus.cc:729] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94af46e728ad452d91785ca419624f59, State: Initialized, Role: FOLLOWER
I20260810 18:50:07.500869 15045 ts_tablet_manager.cc:1397] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51: Time spent bootstrapping tablet: real 0.091s user 0.021s sys 0.042s
I20260810 18:50:07.504655 15042 consensus_queue.cc:260] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.506424 15045 raft_consensus.cc:348] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } }
I20260810 18:50:07.507531 15045 raft_consensus.cc:374] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:07.507951 15045 raft_consensus.cc:729] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b847dd1039674d29a3c811bdfb6c7f51, State: Initialized, Role: FOLLOWER
I20260810 18:50:07.509745 15042 ts_tablet_manager.cc:1428] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59: Time spent starting tablet: real 0.024s user 0.006s sys 0.007s
I20260810 18:50:07.511145 15047 tablet_bootstrap.cc:654] T f14b3e767f224ce2b789dfcfd0414b89 P 49ef0f73074a4b6e9549f68ceb348ae5: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:07.511948 15042 tablet_bootstrap.cc:492] T 9a2e5b7738d54ec99159df35d65b7b32 P 94af46e728ad452d91785ca419624f59: Bootstrap starting.
I20260810 18:50:07.516549 15055 consensus_queue.cc:1035] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:50:07.508980 15045 consensus_queue.cc:260] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } }
I20260810 18:50:07.519759 14982 raft_consensus.cc:2454] T f14b3e767f224ce2b789dfcfd0414b89 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94af46e728ad452d91785ca419624f59 in term 1.
I20260810 18:50:07.523556 14777 leader_election.cc:302] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 94af46e728ad452d91785ca419624f59, b847dd1039674d29a3c811bdfb6c7f51; no voters: 49ef0f73074a4b6e9549f68ceb348ae5
I20260810 18:50:07.526202 15124 raft_consensus.cc:2781] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:07.526923 15124 raft_consensus.cc:686] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59 [term 1 LEADER]: Becoming Leader. State: Replica: 94af46e728ad452d91785ca419624f59, State: Running, Role: LEADER
I20260810 18:50:07.528294 15124 consensus_queue.cc:237] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.532583 15082 raft_consensus.cc:482] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:07.536178 15082 raft_consensus.cc:504] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.536839 15101 raft_consensus.cc:482] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:07.539773 15101 raft_consensus.cc:504] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.544710 15082 leader_election.cc:288] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:07.547355 14843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2a25bed1474a0692bcdfde37ab1d5b" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94af46e728ad452d91785ca419624f59" is_pre_election: true
I20260810 18:50:07.547740 14701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7977ab6121414743ab57535b534ea4f9" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" is_pre_election: true
I20260810 18:50:07.548312 14843 raft_consensus.cc:2454] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49ef0f73074a4b6e9549f68ceb348ae5 in term 0.
I20260810 18:50:07.549070 14701 raft_consensus.cc:2379] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 94af46e728ad452d91785ca419624f59 in current term 1: Already voted for candidate b847dd1039674d29a3c811bdfb6c7f51 in this term.
I20260810 18:50:07.552021 14979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2a25bed1474a0692bcdfde37ab1d5b" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51" is_pre_election: true
I20260810 18:50:07.552429 14982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7977ab6121414743ab57535b534ea4f9" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51" is_pre_election: true
I20260810 18:50:07.554214 14642 leader_election.cc:302] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5, 94af46e728ad452d91785ca419624f59; no voters:
I20260810 18:50:07.555531 15057 raft_consensus.cc:2781] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260810 18:50:07.555720 14642 leader_election.cc:341] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:07.556057 15057 raft_consensus.cc:482] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:07.555801 14777 leader_election.cc:302] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94af46e728ad452d91785ca419624f59; no voters: 49ef0f73074a4b6e9549f68ceb348ae5, b847dd1039674d29a3c811bdfb6c7f51
I20260810 18:50:07.556682 15057 raft_consensus.cc:3037] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:07.546250 15101 leader_election.cc:288] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:07.557245 15082 raft_consensus.cc:3037] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:07.579788 15045 ts_tablet_manager.cc:1428] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51: Time spent starting tablet: real 0.078s user 0.004s sys 0.008s
I20260810 18:50:07.583634 15045 tablet_bootstrap.cc:492] T a8187c40cb114a26b4da50c7bdbd4067 P b847dd1039674d29a3c811bdfb6c7f51: Bootstrap starting.
I20260810 18:50:07.587321 15079 raft_consensus.cc:482] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:07.589679 15079 raft_consensus.cc:504] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.588464 15047 tablet_bootstrap.cc:492] T f14b3e767f224ce2b789dfcfd0414b89 P 49ef0f73074a4b6e9549f68ceb348ae5: No bootstrap required, opened a new log
I20260810 18:50:07.591516 15042 tablet_bootstrap.cc:654] T 9a2e5b7738d54ec99159df35d65b7b32 P 94af46e728ad452d91785ca419624f59: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:07.591840 15047 ts_tablet_manager.cc:1397] T f14b3e767f224ce2b789dfcfd0414b89 P 49ef0f73074a4b6e9549f68ceb348ae5: Time spent bootstrapping tablet: real 0.131s user 0.007s sys 0.018s
I20260810 18:50:07.596352 14701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef3949841d54dfcb19e1f25582bbeaa" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" is_pre_election: true
I20260810 18:50:07.597481 14701 raft_consensus.cc:2454] T 4ef3949841d54dfcb19e1f25582bbeaa P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b847dd1039674d29a3c811bdfb6c7f51 in term 0.
I20260810 18:50:07.600620 14843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef3949841d54dfcb19e1f25582bbeaa" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94af46e728ad452d91785ca419624f59" is_pre_election: true
I20260810 18:50:07.601612 14843 raft_consensus.cc:2454] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b847dd1039674d29a3c811bdfb6c7f51 in term 0.
I20260810 18:50:07.601786 14911 leader_election.cc:302] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5, b847dd1039674d29a3c811bdfb6c7f51; no voters:
I20260810 18:50:07.602826 15079 leader_election.cc:288] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285)
I20260810 18:50:07.605055 15098 raft_consensus.cc:2781] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:07.605810 15098 raft_consensus.cc:482] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:07.606344 15098 raft_consensus.cc:3037] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:07.607986 15079 consensus_queue.cc:1035] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:50:07.625943 15047 raft_consensus.cc:348] T f14b3e767f224ce2b789dfcfd0414b89 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.627050 15047 raft_consensus.cc:374] T f14b3e767f224ce2b789dfcfd0414b89 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:07.627417 15047 raft_consensus.cc:729] T f14b3e767f224ce2b789dfcfd0414b89 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49ef0f73074a4b6e9549f68ceb348ae5, State: Initialized, Role: FOLLOWER
I20260810 18:50:07.628176 15047 consensus_queue.cc:260] T f14b3e767f224ce2b789dfcfd0414b89 P 49ef0f73074a4b6e9549f68ceb348ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.632654 15045 tablet_bootstrap.cc:654] T a8187c40cb114a26b4da50c7bdbd4067 P b847dd1039674d29a3c811bdfb6c7f51: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:07.635833 15135 raft_consensus.cc:482] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:07.639492 15047 ts_tablet_manager.cc:1428] T f14b3e767f224ce2b789dfcfd0414b89 P 49ef0f73074a4b6e9549f68ceb348ae5: Time spent starting tablet: real 0.047s user 0.009s sys 0.000s
I20260810 18:50:07.639662 15042 tablet_bootstrap.cc:492] T 9a2e5b7738d54ec99159df35d65b7b32 P 94af46e728ad452d91785ca419624f59: No bootstrap required, opened a new log
I20260810 18:50:07.640362 15042 ts_tablet_manager.cc:1397] T 9a2e5b7738d54ec99159df35d65b7b32 P 94af46e728ad452d91785ca419624f59: Time spent bootstrapping tablet: real 0.129s user 0.012s sys 0.014s
I20260810 18:50:07.640599 15047 tablet_bootstrap.cc:492] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5: Bootstrap starting.
I20260810 18:50:07.633397 15098 raft_consensus.cc:504] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.648561 15057 raft_consensus.cc:504] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.652930 15098 leader_election.cc:288] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 election: Requested vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285)
I20260810 18:50:07.641973 15135 raft_consensus.cc:504] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
W20260810 18:50:07.659030 15125 log.cc:924] Time spent T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59: Append to log took a long time: real 0.067s user 0.006s sys 0.001s
I20260810 18:50:07.661093 15057 leader_election.cc:288] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 1 election: Requested vote from peers 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:07.653221 14843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2a25bed1474a0692bcdfde37ab1d5b" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94af46e728ad452d91785ca419624f59"
I20260810 18:50:07.663492 14843 raft_consensus.cc:3037] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:07.668689 15135 leader_election.cc:288] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285)
I20260810 18:50:07.672525 14982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2a25bed1474a0692bcdfde37ab1d5b" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
I20260810 18:50:07.670861 15082 raft_consensus.cc:2726] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 18:50:07.674145 14642 leader_election.cc:341] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:07.680235 15057 raft_consensus.cc:482] T f14b3e767f224ce2b789dfcfd0414b89 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:07.681924 14843 raft_consensus.cc:2454] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49ef0f73074a4b6e9549f68ceb348ae5 in term 1.
I20260810 18:50:07.682523 15042 raft_consensus.cc:348] T 9a2e5b7738d54ec99159df35d65b7b32 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.683228 15042 raft_consensus.cc:374] T 9a2e5b7738d54ec99159df35d65b7b32 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:07.683053 14642 leader_election.cc:302] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5, 94af46e728ad452d91785ca419624f59; no voters: b847dd1039674d29a3c811bdfb6c7f51
I20260810 18:50:07.683494 15042 raft_consensus.cc:729] T 9a2e5b7738d54ec99159df35d65b7b32 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94af46e728ad452d91785ca419624f59, State: Initialized, Role: FOLLOWER
I20260810 18:50:07.684155 15101 raft_consensus.cc:2781] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:07.684108 15042 consensus_queue.cc:260] T 9a2e5b7738d54ec99159df35d65b7b32 P 94af46e728ad452d91785ca419624f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.680832 15057 raft_consensus.cc:504] T f14b3e767f224ce2b789dfcfd0414b89 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.690136 15042 ts_tablet_manager.cc:1428] T 9a2e5b7738d54ec99159df35d65b7b32 P 94af46e728ad452d91785ca419624f59: Time spent starting tablet: real 0.049s user 0.005s sys 0.009s
I20260810 18:50:07.691267 15042 tablet_bootstrap.cc:492] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59: Bootstrap starting.
W20260810 18:50:07.687700 14911 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:45253, user_credentials={real_user=slave}} blocked reactor thread for 51719.4us
I20260810 18:50:07.695497 15057 leader_election.cc:288] T f14b3e767f224ce2b789dfcfd0414b89 P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:07.699905 14982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14b3e767f224ce2b789dfcfd0414b89" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51" is_pre_election: true
I20260810 18:50:07.698632 14843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14b3e767f224ce2b789dfcfd0414b89" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94af46e728ad452d91785ca419624f59" is_pre_election: true
I20260810 18:50:07.700709 14519 catalog_manager.cc:5495] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94af46e728ad452d91785ca419624f59 (127.14.32.194). New cstate: current_term: 1 leader_uuid: "94af46e728ad452d91785ca419624f59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:07.697230 15101 raft_consensus.cc:686] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 LEADER]: Becoming Leader. State: Replica: 49ef0f73074a4b6e9549f68ceb348ae5, State: Running, Role: LEADER
I20260810 18:50:07.755681 14706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef3949841d54dfcb19e1f25582bbeaa" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
I20260810 18:50:07.751585 15082 raft_consensus.cc:482] T 9a2e5b7738d54ec99159df35d65b7b32 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:07.750979 15137 raft_consensus.cc:482] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:07.702952 14982 raft_consensus.cc:2379] T f14b3e767f224ce2b789dfcfd0414b89 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49ef0f73074a4b6e9549f68ceb348ae5 in current term 1: Already voted for candidate 94af46e728ad452d91785ca419624f59 in this term.
I20260810 18:50:07.733037 15057 raft_consensus.cc:482] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:07.769604 15057 raft_consensus.cc:504] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.756347 14840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a2e5b7738d54ec99159df35d65b7b32" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94af46e728ad452d91785ca419624f59" is_pre_election: true
I20260810 18:50:07.772411 15045 tablet_bootstrap.cc:492] T a8187c40cb114a26b4da50c7bdbd4067 P b847dd1039674d29a3c811bdfb6c7f51: No bootstrap required, opened a new log
I20260810 18:50:07.771728 14642 leader_election.cc:302] T f14b3e767f224ce2b789dfcfd0414b89 P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5; no voters: 94af46e728ad452d91785ca419624f59, b847dd1039674d29a3c811bdfb6c7f51
I20260810 18:50:07.766738 14706 raft_consensus.cc:3037] T 4ef3949841d54dfcb19e1f25582bbeaa P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:07.773998 15124 raft_consensus.cc:482] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:07.765162 15082 raft_consensus.cc:504] T 9a2e5b7738d54ec99159df35d65b7b32 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.767865 15101 consensus_queue.cc:237] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.725728 14701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a2e5b7738d54ec99159df35d65b7b32" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" is_pre_election: true
I20260810 18:50:07.767992 15137 raft_consensus.cc:504] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } }
I20260810 18:50:07.709077 14841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef3949841d54dfcb19e1f25582bbeaa" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94af46e728ad452d91785ca419624f59"
I20260810 18:50:07.779276 14840 raft_consensus.cc:2454] T 9a2e5b7738d54ec99159df35d65b7b32 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b847dd1039674d29a3c811bdfb6c7f51 in term 0.
I20260810 18:50:07.774745 15124 raft_consensus.cc:504] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } }
I20260810 18:50:07.779628 15082 leader_election.cc:288] T 9a2e5b7738d54ec99159df35d65b7b32 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
W20260810 18:50:07.756415 14911 consensus_peers.cc:487] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 -> Peer 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285): Couldn't send request to peer 94af46e728ad452d91785ca419624f59. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:50:07.773059 15057 leader_election.cc:288] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:07.782267 14979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a2e5b7738d54ec99159df35d65b7b32" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51" is_pre_election: true
I20260810 18:50:07.782568 15045 ts_tablet_manager.cc:1397] T a8187c40cb114a26b4da50c7bdbd4067 P b847dd1039674d29a3c811bdfb6c7f51: Time spent bootstrapping tablet: real 0.200s user 0.010s sys 0.020s
I20260810 18:50:07.789124 15082 raft_consensus.cc:482] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:07.789245 14701 raft_consensus.cc:2454] T 9a2e5b7738d54ec99159df35d65b7b32 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b847dd1039674d29a3c811bdfb6c7f51 in term 0.
I20260810 18:50:07.787683 14841 raft_consensus.cc:3037] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:07.789021 14979 raft_consensus.cc:2454] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94af46e728ad452d91785ca419624f59 in term 0.
I20260810 18:50:07.798094 15124 leader_election.cc:288] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173), 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253)
I20260810 18:50:07.795876 15082 raft_consensus.cc:504] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.795284 14699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a2e5b7738d54ec99159df35d65b7b32" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" is_pre_election: true
I20260810 18:50:07.803838 14699 raft_consensus.cc:2454] T 9a2e5b7738d54ec99159df35d65b7b32 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94af46e728ad452d91785ca419624f59 in term 0.
I20260810 18:50:07.800149 14981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e52bc7697d4473b77a1bd1f6df04f2" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51" is_pre_election: true
I20260810 18:50:07.800554 15137 leader_election.cc:288] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285), 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253)
I20260810 18:50:07.790977 15144 raft_consensus.cc:3037] T f14b3e767f224ce2b789dfcfd0414b89 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:07.801978 15047 tablet_bootstrap.cc:654] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:07.797251 14982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8187c40cb114a26b4da50c7bdbd4067" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51" is_pre_election: true
I20260810 18:50:07.805816 14981 raft_consensus.cc:2454] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94af46e728ad452d91785ca419624f59 in term 0.
I20260810 18:50:07.806223 14777 leader_election.cc:302] T 9a2e5b7738d54ec99159df35d65b7b32 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94af46e728ad452d91785ca419624f59, b847dd1039674d29a3c811bdfb6c7f51; no voters:
I20260810 18:50:07.803248 14843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8187c40cb114a26b4da50c7bdbd4067" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94af46e728ad452d91785ca419624f59" is_pre_election: true
I20260810 18:50:07.813251 14843 raft_consensus.cc:2454] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49ef0f73074a4b6e9549f68ceb348ae5 in term 0.
I20260810 18:50:07.813890 15082 leader_election.cc:288] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:07.820176 14694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e52bc7697d4473b77a1bd1f6df04f2" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" is_pre_election: true
I20260810 18:50:07.820787 15124 raft_consensus.cc:2781] T 9a2e5b7738d54ec99159df35d65b7b32 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:07.821712 15124 raft_consensus.cc:482] T 9a2e5b7738d54ec99159df35d65b7b32 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:07.821506 14642 leader_election.cc:302] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5, 94af46e728ad452d91785ca419624f59; no voters:
I20260810 18:50:07.822851 14911 leader_election.cc:302] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5, b847dd1039674d29a3c811bdfb6c7f51; no voters:
I20260810 18:50:07.826470 15057 raft_consensus.cc:2781] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:07.827023 15057 raft_consensus.cc:482] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:07.827589 15057 raft_consensus.cc:3037] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:07.832787 14694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e52bc7697d4473b77a1bd1f6df04f2" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" is_pre_election: true
I20260810 18:50:07.828878 15124 raft_consensus.cc:3037] T 9a2e5b7738d54ec99159df35d65b7b32 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:07.835722 15137 raft_consensus.cc:2781] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:07.838105 15137 raft_consensus.cc:482] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:07.838735 15137 raft_consensus.cc:3037] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:07.841518 15042 tablet_bootstrap.cc:654] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59: Neither blocks nor log segments found. Creating new log.
W20260810 18:50:07.841564 14911 leader_election.cc:341] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:07.842020 15079 consensus_queue.cc:1035] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:50:07.845113 14706 raft_consensus.cc:2454] T 4ef3949841d54dfcb19e1f25582bbeaa P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b847dd1039674d29a3c811bdfb6c7f51 in term 1.
I20260810 18:50:07.847730 14843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e52bc7697d4473b77a1bd1f6df04f2" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94af46e728ad452d91785ca419624f59" is_pre_election: true
I20260810 18:50:07.847914 14911 leader_election.cc:302] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5, b847dd1039674d29a3c811bdfb6c7f51; no voters:
I20260810 18:50:07.848608 14843 raft_consensus.cc:2454] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b847dd1039674d29a3c811bdfb6c7f51 in term 0.
I20260810 18:50:07.848863 15079 raft_consensus.cc:2781] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:07.849651 15045 raft_consensus.cc:348] T a8187c40cb114a26b4da50c7bdbd4067 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.850571 15045 raft_consensus.cc:374] T a8187c40cb114a26b4da50c7bdbd4067 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:07.850909 15045 raft_consensus.cc:729] T a8187c40cb114a26b4da50c7bdbd4067 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b847dd1039674d29a3c811bdfb6c7f51, State: Initialized, Role: FOLLOWER
I20260810 18:50:07.851758 15045 consensus_queue.cc:260] T a8187c40cb114a26b4da50c7bdbd4067 P b847dd1039674d29a3c811bdfb6c7f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.853945 14981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8187c40cb114a26b4da50c7bdbd4067" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51" is_pre_election: true
I20260810 18:50:07.855006 14981 raft_consensus.cc:2454] T a8187c40cb114a26b4da50c7bdbd4067 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94af46e728ad452d91785ca419624f59 in term 0.
I20260810 18:50:07.873116 15045 ts_tablet_manager.cc:1428] T a8187c40cb114a26b4da50c7bdbd4067 P b847dd1039674d29a3c811bdfb6c7f51: Time spent starting tablet: real 0.084s user 0.006s sys 0.003s
I20260810 18:50:07.870954 15079 raft_consensus.cc:686] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [term 1 LEADER]: Becoming Leader. State: Replica: b847dd1039674d29a3c811bdfb6c7f51, State: Running, Role: LEADER
I20260810 18:50:07.865640 14706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8187c40cb114a26b4da50c7bdbd4067" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" is_pre_election: true
I20260810 18:50:07.887794 15140 consensus_queue.cc:1035] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260810 18:50:07.871449 15137 raft_consensus.cc:504] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.894148 15079 consensus_queue.cc:237] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.895533 15045 tablet_bootstrap.cc:492] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51: Bootstrap starting.
W20260810 18:50:07.888834 14777 leader_election.cc:341] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:07.896719 14777 leader_election.cc:302] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 94af46e728ad452d91785ca419624f59, b847dd1039674d29a3c811bdfb6c7f51; no voters: 49ef0f73074a4b6e9549f68ceb348ae5
I20260810 18:50:07.898500 15082 raft_consensus.cc:2781] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:07.898958 15082 raft_consensus.cc:482] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:07.899516 15082 raft_consensus.cc:3037] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:07.885567 15057 raft_consensus.cc:504] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.905680 14519 catalog_manager.cc:5495] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193). New cstate: current_term: 1 leader_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:07.913646 15144 raft_consensus.cc:2726] T f14b3e767f224ce2b789dfcfd0414b89 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:50:07.914839 15047 tablet_bootstrap.cc:492] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5: No bootstrap required, opened a new log
I20260810 18:50:07.917213 15047 ts_tablet_manager.cc:1397] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5: Time spent bootstrapping tablet: real 0.277s user 0.017s sys 0.012s
I20260810 18:50:07.912431 14706 raft_consensus.cc:2379] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 94af46e728ad452d91785ca419624f59 in current term 1: Already voted for candidate 49ef0f73074a4b6e9549f68ceb348ae5 in this term.
I20260810 18:50:07.918936 14699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a2e5b7738d54ec99159df35d65b7b32" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
I20260810 18:50:07.919857 14699 raft_consensus.cc:3037] T 9a2e5b7738d54ec99159df35d65b7b32 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:07.920447 15137 leader_election.cc:288] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 election: Requested vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285)
I20260810 18:50:07.899106 14777 leader_election.cc:302] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94af46e728ad452d91785ca419624f59, b847dd1039674d29a3c811bdfb6c7f51; no voters:
I20260810 18:50:07.929476 15042 tablet_bootstrap.cc:492] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59: No bootstrap required, opened a new log
I20260810 18:50:07.928705 14843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8187c40cb114a26b4da50c7bdbd4067" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94af46e728ad452d91785ca419624f59"
I20260810 18:50:07.931021 15042 ts_tablet_manager.cc:1397] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59: Time spent bootstrapping tablet: real 0.240s user 0.025s sys 0.022s
I20260810 18:50:07.932837 15047 raft_consensus.cc:348] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } }
I20260810 18:50:07.933672 15047 raft_consensus.cc:374] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:07.933888 15047 raft_consensus.cc:729] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49ef0f73074a4b6e9549f68ceb348ae5, State: Initialized, Role: FOLLOWER
I20260810 18:50:07.934415 15047 consensus_queue.cc:260] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } }
I20260810 18:50:07.935971 15146 raft_consensus.cc:2781] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:07.936475 14982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8187c40cb114a26b4da50c7bdbd4067" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
I20260810 18:50:07.936723 15146 raft_consensus.cc:482] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:07.938294 14982 raft_consensus.cc:3037] T a8187c40cb114a26b4da50c7bdbd4067 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:07.938645 15146 raft_consensus.cc:3037] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:07.940752 14911 leader_election.cc:302] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 94af46e728ad452d91785ca419624f59, b847dd1039674d29a3c811bdfb6c7f51; no voters: 49ef0f73074a4b6e9549f68ceb348ae5
I20260810 18:50:07.922727 14841 raft_consensus.cc:2454] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b847dd1039674d29a3c811bdfb6c7f51 in term 1.
I20260810 18:50:07.925105 15057 leader_election.cc:288] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 1 election: Requested vote from peers 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:07.943049 15140 raft_consensus.cc:2781] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:07.944635 15082 raft_consensus.cc:504] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } }
I20260810 18:50:07.948998 15140 raft_consensus.cc:482] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:07.950098 15140 raft_consensus.cc:3037] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:07.954720 15047 ts_tablet_manager.cc:1428] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5: Time spent starting tablet: real 0.037s user 0.005s sys 0.005s
I20260810 18:50:07.941219 15124 raft_consensus.cc:504] T 9a2e5b7738d54ec99159df35d65b7b32 P 94af46e728ad452d91785ca419624f59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:07.968729 14981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e52bc7697d4473b77a1bd1f6df04f2" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
I20260810 18:50:07.960232 14840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a2e5b7738d54ec99159df35d65b7b32" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94af46e728ad452d91785ca419624f59"
I20260810 18:50:07.970780 14840 raft_consensus.cc:2379] T 9a2e5b7738d54ec99159df35d65b7b32 P 94af46e728ad452d91785ca419624f59 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b847dd1039674d29a3c811bdfb6c7f51 in current term 1: Already voted for candidate 94af46e728ad452d91785ca419624f59 in this term.
I20260810 18:50:07.970525 14706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e52bc7697d4473b77a1bd1f6df04f2" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
I20260810 18:50:07.971621 14706 raft_consensus.cc:3037] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:07.985826 15124 leader_election.cc:288] T 9a2e5b7738d54ec99159df35d65b7b32 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 1 election: Requested vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:07.959781 15082 leader_election.cc:288] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 1 election: Requested vote from peers b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173), 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253)
I20260810 18:50:07.987543 14982 raft_consensus.cc:2454] T a8187c40cb114a26b4da50c7bdbd4067 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49ef0f73074a4b6e9549f68ceb348ae5 in term 1.
I20260810 18:50:08.001178 14699 raft_consensus.cc:2454] T 9a2e5b7738d54ec99159df35d65b7b32 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b847dd1039674d29a3c811bdfb6c7f51 in term 1.
I20260810 18:50:08.002951 14911 leader_election.cc:302] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5, b847dd1039674d29a3c811bdfb6c7f51; no voters: 94af46e728ad452d91785ca419624f59
I20260810 18:50:07.998368 15042 raft_consensus.cc:348] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:08.012005 15042 raft_consensus.cc:374] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:08.012395 15042 raft_consensus.cc:729] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94af46e728ad452d91785ca419624f59, State: Initialized, Role: FOLLOWER
I20260810 18:50:08.013700 15079 raft_consensus.cc:2781] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:08.025110 15079 raft_consensus.cc:686] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 LEADER]: Becoming Leader. State: Replica: b847dd1039674d29a3c811bdfb6c7f51, State: Running, Role: LEADER
I20260810 18:50:08.026911 15045 tablet_bootstrap.cc:654] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:08.026340 15079 consensus_queue.cc:237] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:08.002574 14706 raft_consensus.cc:2454] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94af46e728ad452d91785ca419624f59 in term 1.
I20260810 18:50:08.026347 14642 leader_election.cc:302] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5, b847dd1039674d29a3c811bdfb6c7f51; no voters:
I20260810 18:50:08.024875 15042 consensus_queue.cc:260] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:08.028963 15057 raft_consensus.cc:2781] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:08.035988 15057 raft_consensus.cc:686] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 LEADER]: Becoming Leader. State: Replica: 49ef0f73074a4b6e9549f68ceb348ae5, State: Running, Role: LEADER
I20260810 18:50:08.036959 15057 consensus_queue.cc:237] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:08.031144 15140 raft_consensus.cc:504] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } }
I20260810 18:50:08.039722 14519 catalog_manager.cc:5495] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 reported cstate change: term changed from 0 to 1, leader changed from <none> to b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195). New cstate: current_term: 1 leader_uuid: "b847dd1039674d29a3c811bdfb6c7f51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } health_report { overall_health: HEALTHY } } }
I20260810 18:50:08.051021 15042 ts_tablet_manager.cc:1428] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59: Time spent starting tablet: real 0.120s user 0.008s sys 0.002s
I20260810 18:50:08.057770 15146 raft_consensus.cc:504] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:08.061967 14843 raft_consensus.cc:2379] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 49ef0f73074a4b6e9549f68ceb348ae5 in current term 1: Already voted for candidate 94af46e728ad452d91785ca419624f59 in this term.
I20260810 18:50:08.062160 14981 raft_consensus.cc:2379] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 94af46e728ad452d91785ca419624f59 in current term 1: Already voted for candidate b847dd1039674d29a3c811bdfb6c7f51 in this term.
I20260810 18:50:08.066123 14694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a2e5b7738d54ec99159df35d65b7b32" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
I20260810 18:50:08.067373 14694 raft_consensus.cc:2379] T 9a2e5b7738d54ec99159df35d65b7b32 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 94af46e728ad452d91785ca419624f59 in current term 1: Already voted for candidate b847dd1039674d29a3c811bdfb6c7f51 in this term.
I20260810 18:50:08.065078 15146 leader_election.cc:288] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 1 election: Requested vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:08.072898 14982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a2e5b7738d54ec99159df35d65b7b32" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
I20260810 18:50:08.082191 14777 leader_election.cc:302] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5, 94af46e728ad452d91785ca419624f59; no voters:
I20260810 18:50:08.085321 14706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8187c40cb114a26b4da50c7bdbd4067" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
I20260810 18:50:08.087366 14982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8187c40cb114a26b4da50c7bdbd4067" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
I20260810 18:50:08.088526 14982 raft_consensus.cc:2379] T a8187c40cb114a26b4da50c7bdbd4067 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 94af46e728ad452d91785ca419624f59 in current term 1: Already voted for candidate 49ef0f73074a4b6e9549f68ceb348ae5 in this term.
I20260810 18:50:08.089457 15146 raft_consensus.cc:2781] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:08.090703 15146 raft_consensus.cc:686] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 1 LEADER]: Becoming Leader. State: Replica: 94af46e728ad452d91785ca419624f59, State: Running, Role: LEADER
I20260810 18:50:08.091698 14706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e52bc7697d4473b77a1bd1f6df04f2" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
I20260810 18:50:08.092936 14706 raft_consensus.cc:2379] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b847dd1039674d29a3c811bdfb6c7f51 in current term 1: Already voted for candidate 94af46e728ad452d91785ca419624f59 in this term.
I20260810 18:50:08.092079 15146 consensus_queue.cc:237] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } }
I20260810 18:50:08.098270 15140 leader_election.cc:288] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 election: Requested vote from peers 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285), 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253)
I20260810 18:50:08.101321 14777 leader_election.cc:302] T 9a2e5b7738d54ec99159df35d65b7b32 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94af46e728ad452d91785ca419624f59; no voters: 49ef0f73074a4b6e9549f68ceb348ae5, b847dd1039674d29a3c811bdfb6c7f51
I20260810 18:50:08.101567 14519 catalog_manager.cc:5495] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51 reported cstate change: term changed from 0 to 1, leader changed from <none> to b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195). New cstate: current_term: 1 leader_uuid: "b847dd1039674d29a3c811bdfb6c7f51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } health_report { overall_health: HEALTHY } } }
I20260810 18:50:08.103082 15162 raft_consensus.cc:2726] T 9a2e5b7738d54ec99159df35d65b7b32 P 94af46e728ad452d91785ca419624f59 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 18:50:08.115821 14841 raft_consensus.cc:3037] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:08.116025 14843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e52bc7697d4473b77a1bd1f6df04f2" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94af46e728ad452d91785ca419624f59"
I20260810 18:50:08.118233 15045 tablet_bootstrap.cc:492] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51: No bootstrap required, opened a new log
I20260810 18:50:08.118755 15045 ts_tablet_manager.cc:1397] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51: Time spent bootstrapping tablet: real 0.223s user 0.014s sys 0.010s
I20260810 18:50:08.121848 15045 raft_consensus.cc:348] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:08.122586 15045 raft_consensus.cc:374] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:08.122850 15045 raft_consensus.cc:729] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b847dd1039674d29a3c811bdfb6c7f51, State: Initialized, Role: FOLLOWER
I20260810 18:50:08.124991 14777 leader_election.cc:302] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94af46e728ad452d91785ca419624f59; no voters: 49ef0f73074a4b6e9549f68ceb348ae5, b847dd1039674d29a3c811bdfb6c7f51
I20260810 18:50:08.123708 15045 consensus_queue.cc:260] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:08.127313 15163 raft_consensus.cc:2726] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 18:50:08.127041 14518 catalog_manager.cc:5495] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193). New cstate: current_term: 1 leader_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:08.125250 14911 leader_election.cc:302] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b847dd1039674d29a3c811bdfb6c7f51; no voters: 49ef0f73074a4b6e9549f68ceb348ae5, 94af46e728ad452d91785ca419624f59
I20260810 18:50:08.136471 15162 consensus_queue.cc:1035] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:50:08.138680 15045 ts_tablet_manager.cc:1428] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51: Time spent starting tablet: real 0.020s user 0.008s sys 0.000s
I20260810 18:50:08.141245 15124 consensus_queue.cc:1035] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 18:50:08.142270 14600 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260810 18:50:08.143761 15135 raft_consensus.cc:2726] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 18:50:08.217818 14519 catalog_manager.cc:5495] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94af46e728ad452d91785ca419624f59 (127.14.32.194). New cstate: current_term: 1 leader_uuid: "94af46e728ad452d91785ca419624f59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:08.280344 14841 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "69e52bc7697d4473b77a1bd1f6df04f2"
dest_uuid: "94af46e728ad452d91785ca419624f59"
mode: GRACEFUL
new_leader_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
from {username='slave'} at 127.0.0.1:50740
I20260810 18:50:08.281109 14841 raft_consensus.cc:595] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 1 LEADER]: Received request to transfer leadership to TS 49ef0f73074a4b6e9549f68ceb348ae5
I20260810 18:50:08.305474 14981 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7977ab6121414743ab57535b534ea4f9"
dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
mode: GRACEFUL
new_leader_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
from {username='slave'} at 127.0.0.1:49536
I20260810 18:50:08.306340 14981 raft_consensus.cc:595] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 LEADER]: Received request to transfer leadership to TS 49ef0f73074a4b6e9549f68ceb348ae5
I20260810 18:50:08.316011 14981 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "20ae3d07fdc14d4e9be86682e53a7d10"
dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
mode: GRACEFUL
new_leader_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
from {username='slave'} at 127.0.0.1:49536
I20260810 18:50:08.316620 14981 raft_consensus.cc:595] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 LEADER]: Received request to transfer leadership to TS 49ef0f73074a4b6e9549f68ceb348ae5
I20260810 18:50:08.325829 15135 consensus_queue.cc:1035] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:50:08.335671 14981 raft_consensus.cc:3037] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:08.352739 15057 consensus_queue.cc:1035] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:50:08.357435 14694 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a8187c40cb114a26b4da50c7bdbd4067"
dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
mode: GRACEFUL
new_leader_uuid: "94af46e728ad452d91785ca419624f59"
from {username='slave'} at 127.0.0.1:37000
I20260810 18:50:08.358331 14694 raft_consensus.cc:595] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 LEADER]: Received request to transfer leadership to TS 94af46e728ad452d91785ca419624f59
I20260810 18:50:08.365943 14982 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4ef3949841d54dfcb19e1f25582bbeaa"
dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
mode: GRACEFUL
new_leader_uuid: "94af46e728ad452d91785ca419624f59"
from {username='slave'} at 127.0.0.1:49536
I20260810 18:50:08.366828 14982 raft_consensus.cc:595] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [term 1 LEADER]: Received request to transfer leadership to TS 94af46e728ad452d91785ca419624f59
I20260810 18:50:08.375249 14982 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "395eacb81f614569895253d87e09b9ba"
dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
mode: GRACEFUL
new_leader_uuid: "94af46e728ad452d91785ca419624f59"
from {username='slave'} at 127.0.0.1:49536
I20260810 18:50:08.376049 14982 raft_consensus.cc:595] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 1 LEADER]: Received request to transfer leadership to TS 94af46e728ad452d91785ca419624f59
I20260810 18:50:08.380620 15057 consensus_queue.cc:1035] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:50:08.402300 15135 raft_consensus.cc:979] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51: : Instructing follower 94af46e728ad452d91785ca419624f59 to start an election
I20260810 18:50:08.403294 15098 raft_consensus.cc:1067] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [term 1 LEADER]: Signalling peer 94af46e728ad452d91785ca419624f59 to start an election
I20260810 18:50:08.411058 14840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ef3949841d54dfcb19e1f25582bbeaa"
dest_uuid: "94af46e728ad452d91785ca419624f59"
from {username='slave'} at 127.0.0.1:50726
I20260810 18:50:08.411949 14840 raft_consensus.cc:482] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:08.412844 14840 raft_consensus.cc:3037] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:08.436702 15098 consensus_queue.cc:1035] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:50:08.438774 14840 raft_consensus.cc:504] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:08.442167 14840 leader_election.cc:288] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 2 election: Requested vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:08.445712 14706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef3949841d54dfcb19e1f25582bbeaa" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
I20260810 18:50:08.446050 14981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef3949841d54dfcb19e1f25582bbeaa" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
I20260810 18:50:08.446930 14981 raft_consensus.cc:3032] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:50:08.447660 14981 raft_consensus.cc:729] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b847dd1039674d29a3c811bdfb6c7f51, State: Running, Role: LEADER
I20260810 18:50:08.449055 14981 consensus_queue.cc:260] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:08.463212 14981 raft_consensus.cc:3037] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:08.491631 15135 consensus_queue.cc:1035] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:50:08.498977 14981 raft_consensus.cc:2454] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94af46e728ad452d91785ca419624f59 in term 2.
I20260810 18:50:08.504984 14777 leader_election.cc:302] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 94af46e728ad452d91785ca419624f59, b847dd1039674d29a3c811bdfb6c7f51; no voters: 49ef0f73074a4b6e9549f68ceb348ae5
I20260810 18:50:08.506894 15146 raft_consensus.cc:2781] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:50:08.507735 15146 raft_consensus.cc:686] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [term 2 LEADER]: Becoming Leader. State: Replica: 94af46e728ad452d91785ca419624f59, State: Running, Role: LEADER
I20260810 18:50:08.509298 15146 consensus_queue.cc:237] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:08.527333 15055 raft_consensus.cc:979] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51: : Instructing follower 49ef0f73074a4b6e9549f68ceb348ae5 to start an election
I20260810 18:50:08.527877 15055 raft_consensus.cc:1067] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 LEADER]: Signalling peer 49ef0f73074a4b6e9549f68ceb348ae5 to start an election
I20260810 18:50:08.528980 15098 consensus_queue.cc:1035] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:50:08.530480 14706 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "20ae3d07fdc14d4e9be86682e53a7d10"
dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
from {username='slave'} at 127.0.0.1:36982
I20260810 18:50:08.531644 14706 raft_consensus.cc:482] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:08.532213 14706 raft_consensus.cc:3037] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:08.538065 14519 catalog_manager.cc:5495] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 reported cstate change: term changed from 1 to 2, leader changed from b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195) to 94af46e728ad452d91785ca419624f59 (127.14.32.194). New cstate: current_term: 2 leader_uuid: "94af46e728ad452d91785ca419624f59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:08.545284 15124 consensus_queue.cc:1035] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:50:08.543509 14706 raft_consensus.cc:504] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:08.551146 14706 leader_election.cc:288] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 2 election: Requested vote from peers 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:08.552358 14843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20ae3d07fdc14d4e9be86682e53a7d10" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "94af46e728ad452d91785ca419624f59"
I20260810 18:50:08.553323 14843 raft_consensus.cc:3037] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:08.556989 14982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20ae3d07fdc14d4e9be86682e53a7d10" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
I20260810 18:50:08.557960 14982 raft_consensus.cc:3032] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:50:08.558425 14982 raft_consensus.cc:729] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b847dd1039674d29a3c811bdfb6c7f51, State: Running, Role: LEADER
I20260810 18:50:08.559679 14982 consensus_queue.cc:260] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:08.564674 15144 consensus_queue.cc:1035] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:50:08.565248 14982 raft_consensus.cc:3037] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:08.577288 15055 raft_consensus.cc:979] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51: : Instructing follower 94af46e728ad452d91785ca419624f59 to start an election
I20260810 18:50:08.577860 15098 raft_consensus.cc:1067] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 1 LEADER]: Signalling peer 94af46e728ad452d91785ca419624f59 to start an election
I20260810 18:50:08.580583 14841 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "395eacb81f614569895253d87e09b9ba"
dest_uuid: "94af46e728ad452d91785ca419624f59"
from {username='slave'} at 127.0.0.1:50726
I20260810 18:50:08.581274 14841 raft_consensus.cc:482] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:08.581902 14841 raft_consensus.cc:3037] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:08.582154 14982 raft_consensus.cc:2454] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49ef0f73074a4b6e9549f68ceb348ae5 in term 2.
I20260810 18:50:08.584187 14642 leader_election.cc:302] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5, b847dd1039674d29a3c811bdfb6c7f51; no voters:
I20260810 18:50:08.586043 15144 raft_consensus.cc:2781] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:50:08.586849 15144 raft_consensus.cc:686] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 LEADER]: Becoming Leader. State: Replica: 49ef0f73074a4b6e9549f68ceb348ae5, State: Running, Role: LEADER
I20260810 18:50:08.588378 15144 consensus_queue.cc:237] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:08.600224 14843 raft_consensus.cc:2454] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49ef0f73074a4b6e9549f68ceb348ae5 in term 2.
I20260810 18:50:08.606587 15162 raft_consensus.cc:979] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59: : Instructing follower 49ef0f73074a4b6e9549f68ceb348ae5 to start an election
I20260810 18:50:08.607257 15124 raft_consensus.cc:1067] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 1 LEADER]: Signalling peer 49ef0f73074a4b6e9549f68ceb348ae5 to start an election
I20260810 18:50:08.613636 14694 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "69e52bc7697d4473b77a1bd1f6df04f2"
dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
from {username='slave'} at 127.0.0.1:36996
I20260810 18:50:08.614336 14694 raft_consensus.cc:482] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:08.614807 14694 raft_consensus.cc:3037] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:08.615427 14519 catalog_manager.cc:5495] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5 reported cstate change: term changed from 1 to 2, leader changed from b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195) to 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193). New cstate: current_term: 2 leader_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:08.622469 15144 consensus_queue.cc:1035] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:50:08.629601 14694 raft_consensus.cc:504] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } }
I20260810 18:50:08.631621 14706 raft_consensus.cc:1226] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 FOLLOWER]: Rejecting Update request from peer 94af46e728ad452d91785ca419624f59 for earlier term 1. Current term is 2. Ops: []
I20260810 18:50:08.637676 15162 consensus_queue.cc:1035] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260810 18:50:08.640503 15146 consensus_queue.cc:1046] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:50:08.642807 15163 raft_consensus.cc:3032] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:50:08.643447 15163 raft_consensus.cc:729] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 94af46e728ad452d91785ca419624f59, State: Running, Role: LEADER
I20260810 18:50:08.645309 14841 raft_consensus.cc:504] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:08.648752 14841 leader_election.cc:288] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 2 election: Requested vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:08.644903 15163 consensus_queue.cc:260] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } }
I20260810 18:50:08.651685 14706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395eacb81f614569895253d87e09b9ba" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
I20260810 18:50:08.652526 15163 raft_consensus.cc:3037] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:08.652681 14706 raft_consensus.cc:3037] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:08.661972 14694 leader_election.cc:288] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 2 election: Requested vote from peers b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173), 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285)
I20260810 18:50:08.662947 15055 raft_consensus.cc:979] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51: : Instructing follower 49ef0f73074a4b6e9549f68ceb348ae5 to start an election
I20260810 18:50:08.663501 15055 raft_consensus.cc:1067] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 LEADER]: Signalling peer 49ef0f73074a4b6e9549f68ceb348ae5 to start an election
I20260810 18:50:08.664832 14979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e52bc7697d4473b77a1bd1f6df04f2" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
I20260810 18:50:08.665424 14979 raft_consensus.cc:3037] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:08.666857 14840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e52bc7697d4473b77a1bd1f6df04f2" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "94af46e728ad452d91785ca419624f59"
I20260810 18:50:08.669121 14694 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7977ab6121414743ab57535b534ea4f9"
dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
from {username='slave'} at 127.0.0.1:36982
I20260810 18:50:08.670472 14982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395eacb81f614569895253d87e09b9ba" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
I20260810 18:50:08.671384 14982 raft_consensus.cc:3032] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:50:08.671736 14982 raft_consensus.cc:729] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b847dd1039674d29a3c811bdfb6c7f51, State: Running, Role: LEADER
I20260810 18:50:08.674775 14982 consensus_queue.cc:260] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:08.670454 14694 raft_consensus.cc:482] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:08.676743 14694 raft_consensus.cc:3037] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:08.685060 14982 raft_consensus.cc:3037] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:08.691972 15057 raft_consensus.cc:979] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5: : Instructing follower 94af46e728ad452d91785ca419624f59 to start an election
I20260810 18:50:08.693684 15144 raft_consensus.cc:1067] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 LEADER]: Signalling peer 94af46e728ad452d91785ca419624f59 to start an election
I20260810 18:50:08.703148 14843 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a8187c40cb114a26b4da50c7bdbd4067"
dest_uuid: "94af46e728ad452d91785ca419624f59"
from {username='slave'} at 127.0.0.1:50730
I20260810 18:50:08.703863 14843 raft_consensus.cc:482] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:08.704360 14843 raft_consensus.cc:3037] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:08.706420 14706 raft_consensus.cc:2454] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94af46e728ad452d91785ca419624f59 in term 2.
I20260810 18:50:08.710402 14694 raft_consensus.cc:504] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:08.715288 14979 raft_consensus.cc:2454] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49ef0f73074a4b6e9549f68ceb348ae5 in term 2.
I20260810 18:50:08.716584 14978 raft_consensus.cc:1226] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Rejecting Update request from peer 94af46e728ad452d91785ca419624f59 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260810 18:50:08.716543 14839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7977ab6121414743ab57535b534ea4f9" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "94af46e728ad452d91785ca419624f59"
I20260810 18:50:08.721450 14839 raft_consensus.cc:3037] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:08.721587 14642 leader_election.cc:302] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5, b847dd1039674d29a3c811bdfb6c7f51; no voters:
I20260810 18:50:08.723248 15057 raft_consensus.cc:2781] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:50:08.727495 14843 raft_consensus.cc:504] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:08.732023 15057 raft_consensus.cc:686] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 LEADER]: Becoming Leader. State: Replica: 49ef0f73074a4b6e9549f68ceb348ae5, State: Running, Role: LEADER
I20260810 18:50:08.735832 14777 leader_election.cc:302] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5, 94af46e728ad452d91785ca419624f59; no voters:
I20260810 18:50:08.736469 15057 consensus_queue.cc:237] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } }
I20260810 18:50:08.738430 15163 raft_consensus.cc:2781] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:50:08.735065 14843 leader_election.cc:288] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 2 election: Requested vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:08.737677 14979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8187c40cb114a26b4da50c7bdbd4067" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
I20260810 18:50:08.739224 15163 raft_consensus.cc:686] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 2 LEADER]: Becoming Leader. State: Replica: 94af46e728ad452d91785ca419624f59, State: Running, Role: LEADER
I20260810 18:50:08.739400 14979 raft_consensus.cc:3037] T a8187c40cb114a26b4da50c7bdbd4067 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:08.740679 15163 consensus_queue.cc:237] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:08.746650 14706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8187c40cb114a26b4da50c7bdbd4067" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
I20260810 18:50:08.750289 14982 raft_consensus.cc:2454] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94af46e728ad452d91785ca419624f59 in term 2.
I20260810 18:50:08.747597 14706 raft_consensus.cc:3032] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:50:08.753713 14706 raft_consensus.cc:729] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 49ef0f73074a4b6e9549f68ceb348ae5, State: Running, Role: LEADER
I20260810 18:50:08.755192 14979 raft_consensus.cc:2454] T a8187c40cb114a26b4da50c7bdbd4067 P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94af46e728ad452d91785ca419624f59 in term 2.
I20260810 18:50:08.755259 14840 raft_consensus.cc:2454] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49ef0f73074a4b6e9549f68ceb348ae5 in term 2.
I20260810 18:50:08.755314 14978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7977ab6121414743ab57535b534ea4f9" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
I20260810 18:50:08.761840 14978 raft_consensus.cc:3032] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:50:08.762305 14978 raft_consensus.cc:729] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b847dd1039674d29a3c811bdfb6c7f51, State: Running, Role: LEADER
I20260810 18:50:08.763609 14978 consensus_queue.cc:260] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:08.764559 14839 raft_consensus.cc:2454] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49ef0f73074a4b6e9549f68ceb348ae5 in term 2.
I20260810 18:50:08.766279 14642 leader_election.cc:302] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5, 94af46e728ad452d91785ca419624f59; no voters:
I20260810 18:50:08.765483 14978 raft_consensus.cc:3037] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:08.767560 15057 raft_consensus.cc:2781] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:50:08.768901 15057 raft_consensus.cc:686] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 LEADER]: Becoming Leader. State: Replica: 49ef0f73074a4b6e9549f68ceb348ae5, State: Running, Role: LEADER
I20260810 18:50:08.756289 14694 leader_election.cc:288] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 2 election: Requested vote from peers 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:08.770177 15057 consensus_queue.cc:237] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:08.757352 14706 consensus_queue.cc:260] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:08.760804 14777 leader_election.cc:302] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94af46e728ad452d91785ca419624f59, b847dd1039674d29a3c811bdfb6c7f51; no voters:
I20260810 18:50:08.774981 15163 raft_consensus.cc:2781] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:50:08.776114 15163 raft_consensus.cc:686] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [term 2 LEADER]: Becoming Leader. State: Replica: 94af46e728ad452d91785ca419624f59, State: Running, Role: LEADER
I20260810 18:50:08.772522 14519 catalog_manager.cc:5495] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 reported cstate change: term changed from 1 to 2, leader changed from b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195) to 94af46e728ad452d91785ca419624f59 (127.14.32.194). New cstate: current_term: 2 leader_uuid: "94af46e728ad452d91785ca419624f59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:08.777665 15163 consensus_queue.cc:237] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:08.781564 14978 raft_consensus.cc:2454] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49ef0f73074a4b6e9549f68ceb348ae5 in term 2.
I20260810 18:50:08.778602 14706 raft_consensus.cc:3037] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:08.796469 14706 raft_consensus.cc:2454] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94af46e728ad452d91785ca419624f59 in term 2.
I20260810 18:50:08.803272 14519 catalog_manager.cc:5495] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 reported cstate change: term changed from 1 to 2, leader changed from 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193) to 94af46e728ad452d91785ca419624f59 (127.14.32.194). New cstate: current_term: 2 leader_uuid: "94af46e728ad452d91785ca419624f59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:08.812651 14518 catalog_manager.cc:5495] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 reported cstate change: term changed from 1 to 2, leader changed from 94af46e728ad452d91785ca419624f59 (127.14.32.194) to 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193). New cstate: current_term: 2 leader_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } health_report { overall_health: HEALTHY } } }
I20260810 18:50:08.831838 14518 catalog_manager.cc:5495] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 reported cstate change: term changed from 1 to 2, leader changed from b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195) to 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193). New cstate: current_term: 2 leader_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:08.948550 14706 raft_consensus.cc:3037] T 4ef3949841d54dfcb19e1f25582bbeaa P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:08.956054 14706 raft_consensus.cc:1261] T 4ef3949841d54dfcb19e1f25582bbeaa P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 FOLLOWER]: Refusing update from remote peer 94af46e728ad452d91785ca419624f59: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:08.957631 15163 consensus_queue.cc:1035] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:50:08.968421 14978 raft_consensus.cc:1261] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Refusing update from remote peer 94af46e728ad452d91785ca419624f59: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:08.970640 15146 consensus_queue.cc:1035] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:50:09.148872 14978 raft_consensus.cc:1261] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Refusing update from remote peer 49ef0f73074a4b6e9549f68ceb348ae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:09.150421 15057 consensus_queue.cc:1035] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:50:09.153631 14840 raft_consensus.cc:1261] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Refusing update from remote peer 49ef0f73074a4b6e9549f68ceb348ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:09.155189 15196 consensus_queue.cc:1035] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:50:09.171926 14979 raft_consensus.cc:1261] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Refusing update from remote peer 49ef0f73074a4b6e9549f68ceb348ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:09.180132 15154 consensus_queue.cc:1035] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:50:09.192318 14979 raft_consensus.cc:1261] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Refusing update from remote peer 94af46e728ad452d91785ca419624f59: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:09.194974 15163 consensus_queue.cc:1035] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:50:09.209128 14979 raft_consensus.cc:1261] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Refusing update from remote peer 49ef0f73074a4b6e9549f68ceb348ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:09.211707 15196 consensus_queue.cc:1035] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:50:09.215642 14840 raft_consensus.cc:1261] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Refusing update from remote peer 49ef0f73074a4b6e9549f68ceb348ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:09.218124 15196 consensus_queue.cc:1035] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:50:09.221071 14979 raft_consensus.cc:1261] T a8187c40cb114a26b4da50c7bdbd4067 P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Refusing update from remote peer 94af46e728ad452d91785ca419624f59: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:09.226581 14839 raft_consensus.cc:1261] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Refusing update from remote peer 49ef0f73074a4b6e9549f68ceb348ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:09.232705 15144 consensus_queue.cc:1035] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:50:09.227700 15146 consensus_queue.cc:1035] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260810 18:50:09.241772 14706 raft_consensus.cc:1261] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 FOLLOWER]: Refusing update from remote peer 94af46e728ad452d91785ca419624f59: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:09.245788 15146 consensus_queue.cc:1035] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:50:09.264611 14706 raft_consensus.cc:1261] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 FOLLOWER]: Refusing update from remote peer 94af46e728ad452d91785ca419624f59: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:09.269716 15162 consensus_queue.cc:1035] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:50:11.412460 14467 auto_leader_rebalancer-test.cc:269] The leader distribution is
I20260810 18:50:11.413518 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:11.438783 14840 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "395eacb81f614569895253d87e09b9ba"
dest_uuid: "94af46e728ad452d91785ca419624f59"
mode: GRACEFUL
new_leader_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
from {username='slave'} at 127.0.0.1:50748
I20260810 18:50:11.439524 14840 raft_consensus.cc:595] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 2 LEADER]: Received request to transfer leadership to TS b847dd1039674d29a3c811bdfb6c7f51
I20260810 18:50:11.450882 14706 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cd2a25bed1474a0692bcdfde37ab1d5b"
dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
mode: GRACEFUL
new_leader_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
from {username='slave'} at 127.0.0.1:37008
I20260810 18:50:11.451438 14706 raft_consensus.cc:595] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 LEADER]: Received request to transfer leadership to TS b847dd1039674d29a3c811bdfb6c7f51
I20260810 18:50:11.452502 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260810 18:50:11.452978 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:11.713963 15196 raft_consensus.cc:979] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5: : Instructing follower b847dd1039674d29a3c811bdfb6c7f51 to start an election
I20260810 18:50:11.714411 15216 raft_consensus.cc:1067] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 LEADER]: Signalling peer b847dd1039674d29a3c811bdfb6c7f51 to start an election
I20260810 18:50:11.715859 14979 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cd2a25bed1474a0692bcdfde37ab1d5b"
dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
from {username='slave'} at 127.0.0.1:49518
I20260810 18:50:11.716284 14979 raft_consensus.cc:482] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:11.716528 14979 raft_consensus.cc:3037] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:11.720870 14979 raft_consensus.cc:504] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:11.722572 14979 leader_election.cc:288] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 2 election: Requested vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285)
I20260810 18:50:11.723237 14706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2a25bed1474a0692bcdfde37ab1d5b" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
I20260810 18:50:11.723569 14840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2a25bed1474a0692bcdfde37ab1d5b" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "94af46e728ad452d91785ca419624f59"
I20260810 18:50:11.723871 14706 raft_consensus.cc:3032] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:50:11.724167 14840 raft_consensus.cc:3037] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:11.724221 14706 raft_consensus.cc:729] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 49ef0f73074a4b6e9549f68ceb348ae5, State: Running, Role: LEADER
I20260810 18:50:11.725078 14706 consensus_queue.cc:260] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:11.726099 14706 raft_consensus.cc:3037] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:11.729214 14840 raft_consensus.cc:2454] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b847dd1039674d29a3c811bdfb6c7f51 in term 2.
I20260810 18:50:11.730157 14911 leader_election.cc:302] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94af46e728ad452d91785ca419624f59, b847dd1039674d29a3c811bdfb6c7f51; no voters:
I20260810 18:50:11.730260 14706 raft_consensus.cc:2454] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b847dd1039674d29a3c811bdfb6c7f51 in term 2.
I20260810 18:50:11.730865 15135 raft_consensus.cc:2781] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:50:11.732002 15135 raft_consensus.cc:686] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [term 2 LEADER]: Becoming Leader. State: Replica: b847dd1039674d29a3c811bdfb6c7f51, State: Running, Role: LEADER
I20260810 18:50:11.732739 15135 consensus_queue.cc:237] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:11.738904 14518 catalog_manager.cc:5495] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 reported cstate change: term changed from 1 to 2, leader changed from 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193) to b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195). New cstate: current_term: 2 leader_uuid: "b847dd1039674d29a3c811bdfb6c7f51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } health_report { overall_health: HEALTHY } } }
I20260810 18:50:11.743917 15220 raft_consensus.cc:979] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59: : Instructing follower b847dd1039674d29a3c811bdfb6c7f51 to start an election
I20260810 18:50:11.744357 15162 raft_consensus.cc:1067] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 2 LEADER]: Signalling peer b847dd1039674d29a3c811bdfb6c7f51 to start an election
I20260810 18:50:11.746004 14979 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "395eacb81f614569895253d87e09b9ba"
dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
from {username='slave'} at 127.0.0.1:49534
I20260810 18:50:11.746536 14979 raft_consensus.cc:482] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:11.746981 14979 raft_consensus.cc:3037] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:11.752144 14979 raft_consensus.cc:504] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:11.753580 14979 leader_election.cc:288] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 3 election: Requested vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285)
I20260810 18:50:11.754238 14706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395eacb81f614569895253d87e09b9ba" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
I20260810 18:50:11.754663 14840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395eacb81f614569895253d87e09b9ba" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "94af46e728ad452d91785ca419624f59"
I20260810 18:50:11.754824 14706 raft_consensus.cc:3037] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:11.755297 14840 raft_consensus.cc:3032] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:50:11.755627 14840 raft_consensus.cc:729] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 94af46e728ad452d91785ca419624f59, State: Running, Role: LEADER
I20260810 18:50:11.756361 14840 consensus_queue.cc:260] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:11.757553 14840 raft_consensus.cc:3037] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:11.762475 14706 raft_consensus.cc:2454] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b847dd1039674d29a3c811bdfb6c7f51 in term 3.
I20260810 18:50:11.764091 14911 leader_election.cc:302] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5, b847dd1039674d29a3c811bdfb6c7f51; no voters:
I20260810 18:50:11.765365 15135 raft_consensus.cc:2781] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:50:11.765914 15135 raft_consensus.cc:686] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 3 LEADER]: Becoming Leader. State: Replica: b847dd1039674d29a3c811bdfb6c7f51, State: Running, Role: LEADER
I20260810 18:50:11.766413 14840 raft_consensus.cc:2454] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b847dd1039674d29a3c811bdfb6c7f51 in term 3.
I20260810 18:50:11.767011 15135 consensus_queue.cc:237] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:11.776647 14518 catalog_manager.cc:5495] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 reported cstate change: term changed from 2 to 3, leader changed from 94af46e728ad452d91785ca419624f59 (127.14.32.194) to b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195). New cstate: current_term: 3 leader_uuid: "b847dd1039674d29a3c811bdfb6c7f51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } health_report { overall_health: HEALTHY } } }
I20260810 18:50:12.196071 14840 raft_consensus.cc:1261] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Refusing update from remote peer b847dd1039674d29a3c811bdfb6c7f51: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:12.197580 15135 consensus_queue.cc:1035] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:50:12.204332 14694 raft_consensus.cc:1261] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [term 3 FOLLOWER]: Refusing update from remote peer b847dd1039674d29a3c811bdfb6c7f51: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:12.206918 15135 consensus_queue.cc:1035] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:50:12.223469 14706 raft_consensus.cc:1261] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 FOLLOWER]: Refusing update from remote peer b847dd1039674d29a3c811bdfb6c7f51: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:12.226536 15217 consensus_queue.cc:1035] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:50:12.235917 14840 raft_consensus.cc:1261] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 3 FOLLOWER]: Refusing update from remote peer b847dd1039674d29a3c811bdfb6c7f51: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:12.239204 15135 consensus_queue.cc:1035] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:50:12.453827 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:12.460857 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:12.461274 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:13.462103 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:13.466885 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:13.467249 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:15.212558 15252 raft_consensus.cc:482] T f14b3e767f224ce2b789dfcfd0414b89 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 94af46e728ad452d91785ca419624f59)
I20260810 18:50:15.213902 15252 raft_consensus.cc:504] T f14b3e767f224ce2b789dfcfd0414b89 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:15.219187 15252 leader_election.cc:288] T f14b3e767f224ce2b789dfcfd0414b89 P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:15.322652 15247 raft_consensus.cc:482] T a8187c40cb114a26b4da50c7bdbd4067 P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 94af46e728ad452d91785ca419624f59)
I20260810 18:50:15.323318 15247 raft_consensus.cc:504] T a8187c40cb114a26b4da50c7bdbd4067 P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:15.325686 15247 leader_election.cc:288] T a8187c40cb114a26b4da50c7bdbd4067 P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285)
I20260810 18:50:15.326548 14706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8187c40cb114a26b4da50c7bdbd4067" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" is_pre_election: true
I20260810 18:50:15.339876 14978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14b3e767f224ce2b789dfcfd0414b89" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51" is_pre_election: true
I20260810 18:50:15.342111 15247 raft_consensus.cc:482] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 49ef0f73074a4b6e9549f68ceb348ae5)
I20260810 18:50:15.343098 15247 raft_consensus.cc:504] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:15.349174 15247 leader_election.cc:288] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285)
I20260810 18:50:15.350001 14706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20ae3d07fdc14d4e9be86682e53a7d10" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" is_pre_election: true
I20260810 18:50:15.352957 14840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14b3e767f224ce2b789dfcfd0414b89" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "94af46e728ad452d91785ca419624f59" is_pre_election: true
I20260810 18:50:15.350749 15254 raft_consensus.cc:482] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 49ef0f73074a4b6e9549f68ceb348ae5)
I20260810 18:50:15.354451 15254 raft_consensus.cc:504] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:15.356463 14642 leader_election.cc:302] T f14b3e767f224ce2b789dfcfd0414b89 P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5; no voters: 94af46e728ad452d91785ca419624f59, b847dd1039674d29a3c811bdfb6c7f51
I20260810 18:50:15.357887 15252 raft_consensus.cc:2726] T f14b3e767f224ce2b789dfcfd0414b89 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:50:15.362730 15258 raft_consensus.cc:482] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 49ef0f73074a4b6e9549f68ceb348ae5)
I20260810 18:50:15.365020 15254 leader_election.cc:288] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:15.373133 14843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8187c40cb114a26b4da50c7bdbd4067" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "94af46e728ad452d91785ca419624f59" is_pre_election: true
I20260810 18:50:15.374706 14840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20ae3d07fdc14d4e9be86682e53a7d10" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "94af46e728ad452d91785ca419624f59" is_pre_election: true
I20260810 18:50:15.376272 14706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7977ab6121414743ab57535b534ea4f9" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" is_pre_election: true
I20260810 18:50:15.378046 14979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7977ab6121414743ab57535b534ea4f9" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51" is_pre_election: true
I20260810 18:50:15.382755 15258 raft_consensus.cc:504] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:15.384692 14911 leader_election.cc:302] T a8187c40cb114a26b4da50c7bdbd4067 P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b847dd1039674d29a3c811bdfb6c7f51; no voters: 49ef0f73074a4b6e9549f68ceb348ae5, 94af46e728ad452d91785ca419624f59
I20260810 18:50:15.387144 15251 raft_consensus.cc:2726] T a8187c40cb114a26b4da50c7bdbd4067 P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:50:15.388342 14777 leader_election.cc:302] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94af46e728ad452d91785ca419624f59; no voters: 49ef0f73074a4b6e9549f68ceb348ae5, b847dd1039674d29a3c811bdfb6c7f51
I20260810 18:50:15.389271 15259 raft_consensus.cc:2726] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:50:15.389195 14911 leader_election.cc:302] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b847dd1039674d29a3c811bdfb6c7f51; no voters: 49ef0f73074a4b6e9549f68ceb348ae5, 94af46e728ad452d91785ca419624f59
I20260810 18:50:15.390038 15251 raft_consensus.cc:2726] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:50:15.392328 15258 leader_election.cc:288] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:15.393375 14694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20ae3d07fdc14d4e9be86682e53a7d10" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" is_pre_election: true
I20260810 18:50:15.393460 14978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20ae3d07fdc14d4e9be86682e53a7d10" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51" is_pre_election: true
I20260810 18:50:15.394917 14777 leader_election.cc:302] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94af46e728ad452d91785ca419624f59; no voters: 49ef0f73074a4b6e9549f68ceb348ae5, b847dd1039674d29a3c811bdfb6c7f51
I20260810 18:50:15.395562 15258 raft_consensus.cc:2726] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:50:15.554459 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:15.559666 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:15.560096 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260810 18:50:16.323830 15011 debug-util.cc:398] Leaking SignalData structure 0x7b0800174060 after lost signal to thread 14470
W20260810 18:50:16.324620 15011 debug-util.cc:398] Leaking SignalData structure 0x7b0800174740 after lost signal to thread 14572
W20260810 18:50:16.325371 15011 debug-util.cc:398] Leaking SignalData structure 0x7b08001f2040 after lost signal to thread 14746
W20260810 18:50:16.326105 15011 debug-util.cc:398] Leaking SignalData structure 0x7b080011b9c0 after lost signal to thread 14891
W20260810 18:50:16.326784 15011 debug-util.cc:398] Leaking SignalData structure 0x7b08001f3f00 after lost signal to thread 15017
I20260810 18:50:16.561007 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:16.569176 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:16.569695 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:17.570673 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:17.575599 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:17.575996 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:18.644255 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:18.649259 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:18.649658 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260810 18:50:19.270203 14740 debug-util.cc:398] Leaking SignalData structure 0x7b08000ea060 after lost signal to thread 14470
W20260810 18:50:19.271042 14740 debug-util.cc:398] Leaking SignalData structure 0x7b08000ea740 after lost signal to thread 14572
W20260810 18:50:19.271802 14740 debug-util.cc:398] Leaking SignalData structure 0x7b080003f9a0 after lost signal to thread 14746
W20260810 18:50:19.272454 14740 debug-util.cc:398] Leaking SignalData structure 0x7b08001749a0 after lost signal to thread 14891
W20260810 18:50:19.273144 14740 debug-util.cc:398] Leaking SignalData structure 0x7b08001f4500 after lost signal to thread 15017
I20260810 18:50:19.650575 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:19.658607 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:19.659150 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:20.660157 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:20.668169 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:20.668723 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:21.669809 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:21.678030 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:21.678578 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:22.706679 14467 auto_leader_rebalancer-test.cc:288] The leader distribution is
I20260810 18:50:22.711326 14694 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "69e52bc7697d4473b77a1bd1f6df04f2"
dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
mode: GRACEFUL
new_leader_uuid: "94af46e728ad452d91785ca419624f59"
from {username='slave'} at 127.0.0.1:37000
I20260810 18:50:22.712059 14694 raft_consensus.cc:595] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 LEADER]: Received request to transfer leadership to TS 94af46e728ad452d91785ca419624f59
I20260810 18:50:22.716336 14694 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7977ab6121414743ab57535b534ea4f9"
dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
mode: GRACEFUL
new_leader_uuid: "94af46e728ad452d91785ca419624f59"
from {username='slave'} at 127.0.0.1:37000
I20260810 18:50:22.716840 14694 raft_consensus.cc:595] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 LEADER]: Received request to transfer leadership to TS 94af46e728ad452d91785ca419624f59
I20260810 18:50:22.721081 14978 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cd2a25bed1474a0692bcdfde37ab1d5b"
dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
mode: GRACEFUL
new_leader_uuid: "94af46e728ad452d91785ca419624f59"
from {username='slave'} at 127.0.0.1:49536
I20260810 18:50:22.721535 14978 raft_consensus.cc:595] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [term 2 LEADER]: Received request to transfer leadership to TS 94af46e728ad452d91785ca419624f59
I20260810 18:50:22.725698 14694 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "20ae3d07fdc14d4e9be86682e53a7d10"
dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
mode: GRACEFUL
new_leader_uuid: "94af46e728ad452d91785ca419624f59"
from {username='slave'} at 127.0.0.1:37000
I20260810 18:50:22.726109 14694 raft_consensus.cc:595] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 LEADER]: Received request to transfer leadership to TS 94af46e728ad452d91785ca419624f59
I20260810 18:50:22.737479 14978 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "395eacb81f614569895253d87e09b9ba"
dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
mode: GRACEFUL
new_leader_uuid: "94af46e728ad452d91785ca419624f59"
from {username='slave'} at 127.0.0.1:49536
I20260810 18:50:22.737996 14978 raft_consensus.cc:595] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 3 LEADER]: Received request to transfer leadership to TS 94af46e728ad452d91785ca419624f59
I20260810 18:50:22.821285 15262 raft_consensus.cc:979] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5: : Instructing follower 94af46e728ad452d91785ca419624f59 to start an election
I20260810 18:50:22.821765 15283 raft_consensus.cc:1067] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 LEADER]: Signalling peer 94af46e728ad452d91785ca419624f59 to start an election
I20260810 18:50:22.823172 14841 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "20ae3d07fdc14d4e9be86682e53a7d10"
dest_uuid: "94af46e728ad452d91785ca419624f59"
from {username='slave'} at 127.0.0.1:50730
I20260810 18:50:22.823655 14841 raft_consensus.cc:482] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:22.823968 14841 raft_consensus.cc:3037] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:22.832870 14841 raft_consensus.cc:504] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:22.835984 14841 leader_election.cc:288] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 3 election: Requested vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:22.836676 14694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20ae3d07fdc14d4e9be86682e53a7d10" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
I20260810 18:50:22.837237 14978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20ae3d07fdc14d4e9be86682e53a7d10" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
I20260810 18:50:22.837563 14694 raft_consensus.cc:3032] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:50:22.837824 14978 raft_consensus.cc:3037] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:22.837858 14694 raft_consensus.cc:729] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 49ef0f73074a4b6e9549f68ceb348ae5, State: Running, Role: LEADER
I20260810 18:50:22.838687 14694 consensus_queue.cc:260] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:22.840312 14694 raft_consensus.cc:3037] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:22.847944 14978 raft_consensus.cc:2454] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94af46e728ad452d91785ca419624f59 in term 3.
I20260810 18:50:22.848981 14694 raft_consensus.cc:2454] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94af46e728ad452d91785ca419624f59 in term 3.
I20260810 18:50:22.849414 14777 leader_election.cc:302] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94af46e728ad452d91785ca419624f59, b847dd1039674d29a3c811bdfb6c7f51; no voters:
I20260810 18:50:22.850163 15273 raft_consensus.cc:2781] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:50:22.851621 15273 raft_consensus.cc:686] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59 [term 3 LEADER]: Becoming Leader. State: Replica: 94af46e728ad452d91785ca419624f59, State: Running, Role: LEADER
I20260810 18:50:22.852613 15273 consensus_queue.cc:237] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:22.862954 14518 catalog_manager.cc:5495] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59 reported cstate change: term changed from 2 to 3, leader changed from 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193) to 94af46e728ad452d91785ca419624f59 (127.14.32.194). New cstate: current_term: 3 leader_uuid: "94af46e728ad452d91785ca419624f59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:22.881816 15262 raft_consensus.cc:979] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5: : Instructing follower 94af46e728ad452d91785ca419624f59 to start an election
I20260810 18:50:22.882493 15283 raft_consensus.cc:1067] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 LEADER]: Signalling peer 94af46e728ad452d91785ca419624f59 to start an election
I20260810 18:50:22.884536 14841 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "69e52bc7697d4473b77a1bd1f6df04f2"
dest_uuid: "94af46e728ad452d91785ca419624f59"
from {username='slave'} at 127.0.0.1:50730
I20260810 18:50:22.885061 14841 raft_consensus.cc:482] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:22.885387 14841 raft_consensus.cc:3037] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:22.890898 14841 raft_consensus.cc:504] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } }
I20260810 18:50:22.892839 14841 leader_election.cc:288] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 3 election: Requested vote from peers b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173), 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253)
I20260810 18:50:22.893950 14978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e52bc7697d4473b77a1bd1f6df04f2" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
I20260810 18:50:22.894410 14694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e52bc7697d4473b77a1bd1f6df04f2" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
I20260810 18:50:22.894820 14978 raft_consensus.cc:3037] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:22.895062 14694 raft_consensus.cc:3032] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:50:22.895540 14694 raft_consensus.cc:729] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 49ef0f73074a4b6e9549f68ceb348ae5, State: Running, Role: LEADER
I20260810 18:50:22.896468 14694 consensus_queue.cc:260] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } }
I20260810 18:50:22.897761 14694 raft_consensus.cc:3037] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:22.903367 14978 raft_consensus.cc:2454] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94af46e728ad452d91785ca419624f59 in term 3.
I20260810 18:50:22.904482 14777 leader_election.cc:302] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94af46e728ad452d91785ca419624f59, b847dd1039674d29a3c811bdfb6c7f51; no voters:
I20260810 18:50:22.905207 14694 raft_consensus.cc:2454] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94af46e728ad452d91785ca419624f59 in term 3.
I20260810 18:50:22.905297 15273 raft_consensus.cc:2781] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:50:22.906066 15273 raft_consensus.cc:686] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 3 LEADER]: Becoming Leader. State: Replica: 94af46e728ad452d91785ca419624f59, State: Running, Role: LEADER
I20260810 18:50:22.907429 15273 consensus_queue.cc:237] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } }
I20260810 18:50:22.915696 14518 catalog_manager.cc:5495] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 reported cstate change: term changed from 2 to 3, leader changed from 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193) to 94af46e728ad452d91785ca419624f59 (127.14.32.194). New cstate: current_term: 3 leader_uuid: "94af46e728ad452d91785ca419624f59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:22.969327 15262 raft_consensus.cc:979] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5: : Instructing follower 94af46e728ad452d91785ca419624f59 to start an election
I20260810 18:50:22.969856 15283 raft_consensus.cc:1067] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 LEADER]: Signalling peer 94af46e728ad452d91785ca419624f59 to start an election
I20260810 18:50:22.971732 14841 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7977ab6121414743ab57535b534ea4f9"
dest_uuid: "94af46e728ad452d91785ca419624f59"
from {username='slave'} at 127.0.0.1:50730
I20260810 18:50:22.972357 14841 raft_consensus.cc:482] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:22.972705 14841 raft_consensus.cc:3037] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:22.978938 14841 raft_consensus.cc:504] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:22.981127 14841 leader_election.cc:288] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 3 election: Requested vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:22.982156 14694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7977ab6121414743ab57535b534ea4f9" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
I20260810 18:50:22.982774 14694 raft_consensus.cc:3032] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:50:22.982575 14978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7977ab6121414743ab57535b534ea4f9" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
I20260810 18:50:22.983078 14694 raft_consensus.cc:729] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 49ef0f73074a4b6e9549f68ceb348ae5, State: Running, Role: LEADER
I20260810 18:50:22.983642 14978 raft_consensus.cc:3037] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:22.984122 14694 consensus_queue.cc:260] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:22.984975 14694 raft_consensus.cc:3037] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:22.989037 14694 raft_consensus.cc:2454] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94af46e728ad452d91785ca419624f59 in term 3.
I20260810 18:50:22.989869 14978 raft_consensus.cc:2454] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94af46e728ad452d91785ca419624f59 in term 3.
I20260810 18:50:22.989966 14777 leader_election.cc:302] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5, 94af46e728ad452d91785ca419624f59; no voters:
I20260810 18:50:22.990774 15273 raft_consensus.cc:2781] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:50:22.991484 15273 raft_consensus.cc:686] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [term 3 LEADER]: Becoming Leader. State: Replica: 94af46e728ad452d91785ca419624f59, State: Running, Role: LEADER
I20260810 18:50:22.992746 15273 consensus_queue.cc:237] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:23.001730 14518 catalog_manager.cc:5495] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 reported cstate change: term changed from 2 to 3, leader changed from 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193) to 94af46e728ad452d91785ca419624f59 (127.14.32.194). New cstate: current_term: 3 leader_uuid: "94af46e728ad452d91785ca419624f59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:23.026116 15267 raft_consensus.cc:979] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51: : Instructing follower 94af46e728ad452d91785ca419624f59 to start an election
I20260810 18:50:23.027190 15293 raft_consensus.cc:1067] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 3 LEADER]: Signalling peer 94af46e728ad452d91785ca419624f59 to start an election
I20260810 18:50:23.028909 14841 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "395eacb81f614569895253d87e09b9ba"
dest_uuid: "94af46e728ad452d91785ca419624f59"
from {username='slave'} at 127.0.0.1:50726
I20260810 18:50:23.029515 14841 raft_consensus.cc:482] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:23.029934 14841 raft_consensus.cc:3037] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:50:23.035989 14841 raft_consensus.cc:504] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:23.038477 14841 leader_election.cc:288] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 4 election: Requested vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:23.039256 14694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395eacb81f614569895253d87e09b9ba" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
I20260810 18:50:23.039877 14694 raft_consensus.cc:3037] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:50:23.040693 15267 raft_consensus.cc:979] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51: : Instructing follower 94af46e728ad452d91785ca419624f59 to start an election
I20260810 18:50:23.040617 14978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395eacb81f614569895253d87e09b9ba" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
I20260810 18:50:23.041415 15293 raft_consensus.cc:1067] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [term 2 LEADER]: Signalling peer 94af46e728ad452d91785ca419624f59 to start an election
I20260810 18:50:23.041610 14978 raft_consensus.cc:3032] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:50:23.042171 14978 raft_consensus.cc:729] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b847dd1039674d29a3c811bdfb6c7f51, State: Running, Role: LEADER
I20260810 18:50:23.043375 14841 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cd2a25bed1474a0692bcdfde37ab1d5b"
dest_uuid: "94af46e728ad452d91785ca419624f59"
from {username='slave'} at 127.0.0.1:50726
I20260810 18:50:23.044175 14841 raft_consensus.cc:482] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:23.043438 14978 consensus_queue.cc:260] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:23.044674 14841 raft_consensus.cc:3037] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:23.045352 14978 raft_consensus.cc:3037] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:50:23.047663 14694 raft_consensus.cc:2454] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94af46e728ad452d91785ca419624f59 in term 4.
I20260810 18:50:23.048856 14777 leader_election.cc:302] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5, 94af46e728ad452d91785ca419624f59; no voters:
I20260810 18:50:23.049957 15273 raft_consensus.cc:2781] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:50:23.050697 15273 raft_consensus.cc:686] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 4 LEADER]: Becoming Leader. State: Replica: 94af46e728ad452d91785ca419624f59, State: Running, Role: LEADER
I20260810 18:50:23.052265 15273 consensus_queue.cc:237] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:23.054339 14978 raft_consensus.cc:2454] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94af46e728ad452d91785ca419624f59 in term 4.
I20260810 18:50:23.063499 14518 catalog_manager.cc:5495] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 reported cstate change: term changed from 3 to 4, leader changed from b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195) to 94af46e728ad452d91785ca419624f59 (127.14.32.194). New cstate: current_term: 4 leader_uuid: "94af46e728ad452d91785ca419624f59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:23.066636 14841 raft_consensus.cc:504] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:23.069778 14841 leader_election.cc:288] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 3 election: Requested vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:23.070874 14694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2a25bed1474a0692bcdfde37ab1d5b" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
I20260810 18:50:23.071597 14694 raft_consensus.cc:3037] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:23.071376 14978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2a25bed1474a0692bcdfde37ab1d5b" candidate_uuid: "94af46e728ad452d91785ca419624f59" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
I20260810 18:50:23.072341 14978 raft_consensus.cc:3032] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:50:23.072844 14978 raft_consensus.cc:729] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b847dd1039674d29a3c811bdfb6c7f51, State: Running, Role: LEADER
I20260810 18:50:23.074215 14978 consensus_queue.cc:260] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:23.076015 14978 raft_consensus.cc:3037] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:23.087509 14694 raft_consensus.cc:2454] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94af46e728ad452d91785ca419624f59 in term 3.
I20260810 18:50:23.087591 14978 raft_consensus.cc:2454] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94af46e728ad452d91785ca419624f59 in term 3.
I20260810 18:50:23.088747 14777 leader_election.cc:302] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5, 94af46e728ad452d91785ca419624f59; no voters:
I20260810 18:50:23.090512 15273 raft_consensus.cc:2781] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:50:23.091207 15273 raft_consensus.cc:686] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [term 3 LEADER]: Becoming Leader. State: Replica: 94af46e728ad452d91785ca419624f59, State: Running, Role: LEADER
I20260810 18:50:23.092904 15273 consensus_queue.cc:237] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:23.103595 14518 catalog_manager.cc:5495] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 reported cstate change: term changed from 2 to 3, leader changed from b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195) to 94af46e728ad452d91785ca419624f59 (127.14.32.194). New cstate: current_term: 3 leader_uuid: "94af46e728ad452d91785ca419624f59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:23.286762 14694 raft_consensus.cc:1261] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 3 FOLLOWER]: Refusing update from remote peer 94af46e728ad452d91785ca419624f59: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:23.287971 15273 consensus_queue.cc:1035] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:50:23.296873 14978 raft_consensus.cc:1261] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [term 3 FOLLOWER]: Refusing update from remote peer 94af46e728ad452d91785ca419624f59: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:23.300076 15273 consensus_queue.cc:1035] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:50:23.358902 14978 raft_consensus.cc:1261] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [term 3 FOLLOWER]: Refusing update from remote peer 94af46e728ad452d91785ca419624f59: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:23.360597 15284 consensus_queue.cc:1035] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:50:23.375669 14694 raft_consensus.cc:1261] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 3 FOLLOWER]: Refusing update from remote peer 94af46e728ad452d91785ca419624f59: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:23.377218 15284 consensus_queue.cc:1035] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:50:23.421262 14978 raft_consensus.cc:1261] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [term 3 FOLLOWER]: Refusing update from remote peer 94af46e728ad452d91785ca419624f59: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:23.422524 15284 consensus_queue.cc:1035] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:50:23.436275 14694 raft_consensus.cc:1261] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 3 FOLLOWER]: Refusing update from remote peer 94af46e728ad452d91785ca419624f59: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:23.440572 15298 consensus_queue.cc:1035] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:50:23.457217 14978 raft_consensus.cc:1261] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 4 FOLLOWER]: Refusing update from remote peer 94af46e728ad452d91785ca419624f59: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260810 18:50:23.458437 15284 consensus_queue.cc:1035] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:50:23.471302 14694 raft_consensus.cc:1261] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [term 4 FOLLOWER]: Refusing update from remote peer 94af46e728ad452d91785ca419624f59: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260810 18:50:23.473685 15273 consensus_queue.cc:1035] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:50:23.555037 14978 raft_consensus.cc:1261] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [term 3 FOLLOWER]: Refusing update from remote peer 94af46e728ad452d91785ca419624f59: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:23.557315 15273 consensus_queue.cc:1035] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:50:23.573671 14694 raft_consensus.cc:1261] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [term 3 FOLLOWER]: Refusing update from remote peer 94af46e728ad452d91785ca419624f59: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:23.575356 15284 consensus_queue.cc:1035] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:50:25.769726 14467 auto_leader_rebalancer-test.cc:303] The leader distribution is
I20260810 18:50:25.770382 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:25.777215 14841 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "395eacb81f614569895253d87e09b9ba"
dest_uuid: "94af46e728ad452d91785ca419624f59"
mode: GRACEFUL
new_leader_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
from {username='slave'} at 127.0.0.1:50748
I20260810 18:50:25.777812 14841 raft_consensus.cc:595] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 4 LEADER]: Received request to transfer leadership to TS 49ef0f73074a4b6e9549f68ceb348ae5
I20260810 18:50:25.780130 14841 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4ef3949841d54dfcb19e1f25582bbeaa"
dest_uuid: "94af46e728ad452d91785ca419624f59"
mode: GRACEFUL
new_leader_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
from {username='slave'} at 127.0.0.1:50748
I20260810 18:50:25.780535 14841 raft_consensus.cc:595] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [term 2 LEADER]: Received request to transfer leadership to TS b847dd1039674d29a3c811bdfb6c7f51
I20260810 18:50:25.782531 14841 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "69e52bc7697d4473b77a1bd1f6df04f2"
dest_uuid: "94af46e728ad452d91785ca419624f59"
mode: GRACEFUL
new_leader_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
from {username='slave'} at 127.0.0.1:50748
I20260810 18:50:25.782938 14841 raft_consensus.cc:595] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 3 LEADER]: Received request to transfer leadership to TS 49ef0f73074a4b6e9549f68ceb348ae5
I20260810 18:50:25.784897 14841 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a8187c40cb114a26b4da50c7bdbd4067"
dest_uuid: "94af46e728ad452d91785ca419624f59"
mode: GRACEFUL
new_leader_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
from {username='slave'} at 127.0.0.1:50748
I20260810 18:50:25.785297 14841 raft_consensus.cc:595] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [term 2 LEADER]: Received request to transfer leadership to TS 49ef0f73074a4b6e9549f68ceb348ae5
I20260810 18:50:25.787154 14841 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cd2a25bed1474a0692bcdfde37ab1d5b"
dest_uuid: "94af46e728ad452d91785ca419624f59"
mode: GRACEFUL
new_leader_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
from {username='slave'} at 127.0.0.1:50748
I20260810 18:50:25.787549 14841 raft_consensus.cc:595] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [term 3 LEADER]: Received request to transfer leadership to TS b847dd1039674d29a3c811bdfb6c7f51
I20260810 18:50:25.788450 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260810 18:50:25.788894 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:25.845046 15273 raft_consensus.cc:979] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59: : Instructing follower 49ef0f73074a4b6e9549f68ceb348ae5 to start an election
I20260810 18:50:25.845564 15319 raft_consensus.cc:1067] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [term 2 LEADER]: Signalling peer 49ef0f73074a4b6e9549f68ceb348ae5 to start an election
I20260810 18:50:25.847105 14706 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a8187c40cb114a26b4da50c7bdbd4067"
dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
from {username='slave'} at 127.0.0.1:36996
I20260810 18:50:25.847610 14706 raft_consensus.cc:482] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:25.847956 14706 raft_consensus.cc:3037] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:25.853159 14706 raft_consensus.cc:504] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:25.855103 14706 leader_election.cc:288] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 3 election: Requested vote from peers 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:25.856161 14841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8187c40cb114a26b4da50c7bdbd4067" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "94af46e728ad452d91785ca419624f59"
I20260810 18:50:25.856572 14978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8187c40cb114a26b4da50c7bdbd4067" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
I20260810 18:50:25.856837 14841 raft_consensus.cc:3032] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:50:25.857110 14978 raft_consensus.cc:3037] T a8187c40cb114a26b4da50c7bdbd4067 P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:25.857179 14841 raft_consensus.cc:729] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 94af46e728ad452d91785ca419624f59, State: Running, Role: LEADER
I20260810 18:50:25.857981 14841 consensus_queue.cc:260] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:25.859297 14841 raft_consensus.cc:3037] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:25.863579 14978 raft_consensus.cc:2454] T a8187c40cb114a26b4da50c7bdbd4067 P b847dd1039674d29a3c811bdfb6c7f51 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49ef0f73074a4b6e9549f68ceb348ae5 in term 3.
I20260810 18:50:25.864832 14642 leader_election.cc:302] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5, b847dd1039674d29a3c811bdfb6c7f51; no voters:
I20260810 18:50:25.865808 14841 raft_consensus.cc:2454] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49ef0f73074a4b6e9549f68ceb348ae5 in term 3.
I20260810 18:50:25.866307 15321 raft_consensus.cc:2781] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:50:25.867331 15321 raft_consensus.cc:686] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 3 LEADER]: Becoming Leader. State: Replica: 49ef0f73074a4b6e9549f68ceb348ae5, State: Running, Role: LEADER
I20260810 18:50:25.868264 15321 consensus_queue.cc:237] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:25.874753 14518 catalog_manager.cc:5495] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 reported cstate change: term changed from 2 to 3, leader changed from 94af46e728ad452d91785ca419624f59 (127.14.32.194) to 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193). New cstate: current_term: 3 leader_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:25.910624 15273 raft_consensus.cc:979] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59: : Instructing follower b847dd1039674d29a3c811bdfb6c7f51 to start an election
I20260810 18:50:25.911028 15319 raft_consensus.cc:1067] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [term 2 LEADER]: Signalling peer b847dd1039674d29a3c811bdfb6c7f51 to start an election
I20260810 18:50:25.912285 14978 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ef3949841d54dfcb19e1f25582bbeaa"
dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
from {username='slave'} at 127.0.0.1:49534
I20260810 18:50:25.912712 14978 raft_consensus.cc:482] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:25.913000 14978 raft_consensus.cc:3037] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:25.917111 14978 raft_consensus.cc:504] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:25.918753 14978 leader_election.cc:288] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 3 election: Requested vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285)
I20260810 18:50:25.919570 14706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef3949841d54dfcb19e1f25582bbeaa" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
I20260810 18:50:25.919799 14841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef3949841d54dfcb19e1f25582bbeaa" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "94af46e728ad452d91785ca419624f59"
I20260810 18:50:25.920051 14706 raft_consensus.cc:3037] T 4ef3949841d54dfcb19e1f25582bbeaa P 49ef0f73074a4b6e9549f68ceb348ae5 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:25.920352 14841 raft_consensus.cc:3032] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:50:25.920608 14841 raft_consensus.cc:729] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 94af46e728ad452d91785ca419624f59, State: Running, Role: LEADER
I20260810 18:50:25.921168 14841 consensus_queue.cc:260] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:25.921998 14841 raft_consensus.cc:3037] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:25.924129 14706 raft_consensus.cc:2454] T 4ef3949841d54dfcb19e1f25582bbeaa P 49ef0f73074a4b6e9549f68ceb348ae5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b847dd1039674d29a3c811bdfb6c7f51 in term 3.
I20260810 18:50:25.925203 14911 leader_election.cc:302] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5, b847dd1039674d29a3c811bdfb6c7f51; no voters:
I20260810 18:50:25.925936 15293 raft_consensus.cc:2781] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:50:25.925944 14841 raft_consensus.cc:2454] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b847dd1039674d29a3c811bdfb6c7f51 in term 3.
I20260810 18:50:25.927237 15293 raft_consensus.cc:686] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [term 3 LEADER]: Becoming Leader. State: Replica: b847dd1039674d29a3c811bdfb6c7f51, State: Running, Role: LEADER
I20260810 18:50:25.928038 15293 consensus_queue.cc:237] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:25.934940 14518 catalog_manager.cc:5495] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 reported cstate change: term changed from 2 to 3, leader changed from 94af46e728ad452d91785ca419624f59 (127.14.32.194) to b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195). New cstate: current_term: 3 leader_uuid: "b847dd1039674d29a3c811bdfb6c7f51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } health_report { overall_health: HEALTHY } } }
I20260810 18:50:26.045816 15273 raft_consensus.cc:979] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59: : Instructing follower 49ef0f73074a4b6e9549f68ceb348ae5 to start an election
I20260810 18:50:26.046458 15319 raft_consensus.cc:1067] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 3 LEADER]: Signalling peer 49ef0f73074a4b6e9549f68ceb348ae5 to start an election
I20260810 18:50:26.047781 14706 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "69e52bc7697d4473b77a1bd1f6df04f2"
dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
from {username='slave'} at 127.0.0.1:36996
I20260810 18:50:26.048177 14706 raft_consensus.cc:482] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:26.048391 14706 raft_consensus.cc:3037] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:50:26.052211 14706 raft_consensus.cc:504] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } }
I20260810 18:50:26.053573 14706 leader_election.cc:288] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 4 election: Requested vote from peers b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173), 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285)
I20260810 18:50:26.054490 14978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e52bc7697d4473b77a1bd1f6df04f2" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
I20260810 18:50:26.054819 14841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e52bc7697d4473b77a1bd1f6df04f2" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "94af46e728ad452d91785ca419624f59"
I20260810 18:50:26.055090 14978 raft_consensus.cc:3037] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:50:26.055352 14841 raft_consensus.cc:3032] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:50:26.055629 14841 raft_consensus.cc:729] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 94af46e728ad452d91785ca419624f59, State: Running, Role: LEADER
I20260810 18:50:26.056318 14841 consensus_queue.cc:260] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } }
I20260810 18:50:26.057442 14841 raft_consensus.cc:3037] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:50:26.060825 14978 raft_consensus.cc:2454] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49ef0f73074a4b6e9549f68ceb348ae5 in term 4.
I20260810 18:50:26.061774 14642 leader_election.cc:302] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5, b847dd1039674d29a3c811bdfb6c7f51; no voters:
I20260810 18:50:26.062492 15321 raft_consensus.cc:2781] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:50:26.062963 15321 raft_consensus.cc:686] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 4 LEADER]: Becoming Leader. State: Replica: 49ef0f73074a4b6e9549f68ceb348ae5, State: Running, Role: LEADER
I20260810 18:50:26.063024 14841 raft_consensus.cc:2454] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49ef0f73074a4b6e9549f68ceb348ae5 in term 4.
I20260810 18:50:26.064008 15321 consensus_queue.cc:237] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } }
I20260810 18:50:26.070860 14518 catalog_manager.cc:5495] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 reported cstate change: term changed from 3 to 4, leader changed from 94af46e728ad452d91785ca419624f59 (127.14.32.194) to 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193). New cstate: current_term: 4 leader_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } health_report { overall_health: HEALTHY } } }
I20260810 18:50:26.106550 15273 raft_consensus.cc:979] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59: : Instructing follower 49ef0f73074a4b6e9549f68ceb348ae5 to start an election
I20260810 18:50:26.107007 15319 raft_consensus.cc:1067] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 4 LEADER]: Signalling peer 49ef0f73074a4b6e9549f68ceb348ae5 to start an election
I20260810 18:50:26.108219 14706 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "395eacb81f614569895253d87e09b9ba"
dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
from {username='slave'} at 127.0.0.1:36996
I20260810 18:50:26.108621 14706 raft_consensus.cc:482] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:26.108853 14706 raft_consensus.cc:3037] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:50:26.112989 14706 raft_consensus.cc:504] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:26.114431 14706 leader_election.cc:288] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 5 election: Requested vote from peers 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285), b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195:42173)
I20260810 18:50:26.115181 14841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395eacb81f614569895253d87e09b9ba" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "94af46e728ad452d91785ca419624f59"
I20260810 18:50:26.115453 14978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395eacb81f614569895253d87e09b9ba" candidate_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
I20260810 18:50:26.115648 14841 raft_consensus.cc:3032] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 4 LEADER]: Stepping down as leader of term 4
I20260810 18:50:26.115947 14841 raft_consensus.cc:729] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 94af46e728ad452d91785ca419624f59, State: Running, Role: LEADER
I20260810 18:50:26.115969 14978 raft_consensus.cc:3037] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:50:26.116565 14841 consensus_queue.cc:260] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:26.117471 14841 raft_consensus.cc:3037] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:50:26.119931 14978 raft_consensus.cc:2454] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49ef0f73074a4b6e9549f68ceb348ae5 in term 5.
I20260810 18:50:26.120747 14642 leader_election.cc:302] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5, b847dd1039674d29a3c811bdfb6c7f51; no voters:
I20260810 18:50:26.121379 15321 raft_consensus.cc:2781] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:50:26.121886 15321 raft_consensus.cc:686] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [term 5 LEADER]: Becoming Leader. State: Replica: 49ef0f73074a4b6e9549f68ceb348ae5, State: Running, Role: LEADER
I20260810 18:50:26.121914 14841 raft_consensus.cc:2454] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49ef0f73074a4b6e9549f68ceb348ae5 in term 5.
I20260810 18:50:26.122833 15321 consensus_queue.cc:237] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:26.129467 14518 catalog_manager.cc:5495] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 reported cstate change: term changed from 4 to 5, leader changed from 94af46e728ad452d91785ca419624f59 (127.14.32.194) to 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193). New cstate: current_term: 5 leader_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:26.280733 15273 raft_consensus.cc:979] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59: : Instructing follower b847dd1039674d29a3c811bdfb6c7f51 to start an election
I20260810 18:50:26.281306 15319 raft_consensus.cc:1067] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [term 3 LEADER]: Signalling peer b847dd1039674d29a3c811bdfb6c7f51 to start an election
I20260810 18:50:26.282577 14979 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cd2a25bed1474a0692bcdfde37ab1d5b"
dest_uuid: "b847dd1039674d29a3c811bdfb6c7f51"
from {username='slave'} at 127.0.0.1:49534
I20260810 18:50:26.282989 14979 raft_consensus.cc:482] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:26.283257 14979 raft_consensus.cc:3037] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:50:26.288169 14979 raft_consensus.cc:504] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:26.289999 14979 leader_election.cc:288] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 4 election: Requested vote from peers 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253), 94af46e728ad452d91785ca419624f59 (127.14.32.194:45285)
I20260810 18:50:26.290838 14841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2a25bed1474a0692bcdfde37ab1d5b" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "94af46e728ad452d91785ca419624f59"
I20260810 18:50:26.290872 14706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2a25bed1474a0692bcdfde37ab1d5b" candidate_uuid: "b847dd1039674d29a3c811bdfb6c7f51" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "49ef0f73074a4b6e9549f68ceb348ae5"
I20260810 18:50:26.291350 14841 raft_consensus.cc:3032] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:50:26.291550 14706 raft_consensus.cc:3037] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:50:26.291630 14841 raft_consensus.cc:729] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 94af46e728ad452d91785ca419624f59, State: Running, Role: LEADER
I20260810 18:50:26.292397 14841 consensus_queue.cc:260] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:26.293234 14841 raft_consensus.cc:3037] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:50:26.295805 14706 raft_consensus.cc:2454] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b847dd1039674d29a3c811bdfb6c7f51 in term 4.
I20260810 18:50:26.296733 14911 leader_election.cc:302] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ef0f73074a4b6e9549f68ceb348ae5, b847dd1039674d29a3c811bdfb6c7f51; no voters:
I20260810 18:50:26.297376 15293 raft_consensus.cc:2781] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:50:26.297814 15293 raft_consensus.cc:686] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [term 4 LEADER]: Becoming Leader. State: Replica: b847dd1039674d29a3c811bdfb6c7f51, State: Running, Role: LEADER
I20260810 18:50:26.297936 14841 raft_consensus.cc:2454] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b847dd1039674d29a3c811bdfb6c7f51 in term 4.
I20260810 18:50:26.298640 15293 consensus_queue.cc:237] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } }
I20260810 18:50:26.305781 14518 catalog_manager.cc:5495] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 reported cstate change: term changed from 3 to 4, leader changed from 94af46e728ad452d91785ca419624f59 (127.14.32.194) to b847dd1039674d29a3c811bdfb6c7f51 (127.14.32.195). New cstate: current_term: 4 leader_uuid: "b847dd1039674d29a3c811bdfb6c7f51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 } health_report { overall_health: HEALTHY } } }
I20260810 18:50:26.313584 14979 raft_consensus.cc:1261] T a8187c40cb114a26b4da50c7bdbd4067 P b847dd1039674d29a3c811bdfb6c7f51 [term 3 FOLLOWER]: Refusing update from remote peer 49ef0f73074a4b6e9549f68ceb348ae5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:26.314786 15321 consensus_queue.cc:1035] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:50:26.326361 14841 raft_consensus.cc:1261] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [term 3 FOLLOWER]: Refusing update from remote peer 49ef0f73074a4b6e9549f68ceb348ae5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:26.328132 15346 consensus_queue.cc:1035] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:50:26.416231 14706 raft_consensus.cc:1261] T 4ef3949841d54dfcb19e1f25582bbeaa P 49ef0f73074a4b6e9549f68ceb348ae5 [term 3 FOLLOWER]: Refusing update from remote peer b847dd1039674d29a3c811bdfb6c7f51: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:26.417457 15293 consensus_queue.cc:1035] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:50:26.426570 14841 raft_consensus.cc:1261] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [term 3 FOLLOWER]: Refusing update from remote peer b847dd1039674d29a3c811bdfb6c7f51: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:26.427966 15293 consensus_queue.cc:1035] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:50:26.454975 14979 raft_consensus.cc:1261] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [term 4 FOLLOWER]: Refusing update from remote peer 49ef0f73074a4b6e9549f68ceb348ae5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260810 18:50:26.456167 15346 consensus_queue.cc:1035] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:50:26.467283 14841 raft_consensus.cc:1261] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 4 FOLLOWER]: Refusing update from remote peer 49ef0f73074a4b6e9549f68ceb348ae5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260810 18:50:26.468560 15346 consensus_queue.cc:1035] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:50:26.716266 14841 raft_consensus.cc:1261] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 5 FOLLOWER]: Refusing update from remote peer 49ef0f73074a4b6e9549f68ceb348ae5: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260810 18:50:26.716266 14979 raft_consensus.cc:1261] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 5 FOLLOWER]: Refusing update from remote peer 49ef0f73074a4b6e9549f68ceb348ae5: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260810 18:50:26.717799 15355 consensus_queue.cc:1035] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b847dd1039674d29a3c811bdfb6c7f51" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 42173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260810 18:50:26.718463 15346 consensus_queue.cc:1035] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260810 18:50:26.789790 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:26.794610 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:26.794951 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:26.817047 14706 raft_consensus.cc:1261] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [term 4 FOLLOWER]: Refusing update from remote peer b847dd1039674d29a3c811bdfb6c7f51: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260810 18:50:26.818112 15293 consensus_queue.cc:1035] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49ef0f73074a4b6e9549f68ceb348ae5" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:50:26.825807 14841 raft_consensus.cc:1261] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [term 4 FOLLOWER]: Refusing update from remote peer b847dd1039674d29a3c811bdfb6c7f51: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260810 18:50:26.827481 15331 consensus_queue.cc:1035] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94af46e728ad452d91785ca419624f59" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:50:27.952133 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:27.957614 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:27.958020 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260810 18:50:28.234232 14564 debug-util.cc:398] Leaking SignalData structure 0x7b0800060060 after lost signal to thread 14470
W20260810 18:50:28.235143 14564 debug-util.cc:398] Leaking SignalData structure 0x7b0800060740 after lost signal to thread 14572
W20260810 18:50:28.235963 14564 debug-util.cc:398] Leaking SignalData structure 0x7b080020cb60 after lost signal to thread 14746
W20260810 18:50:28.236739 14564 debug-util.cc:398] Leaking SignalData structure 0x7b08001fe080 after lost signal to thread 14891
W20260810 18:50:28.237444 14564 debug-util.cc:398] Leaking SignalData structure 0x7b08001ed0e0 after lost signal to thread 15017
I20260810 18:50:28.958763 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:28.963697 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:28.964093 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:29.964982 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:29.969637 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:29.970081 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:30.971051 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:30.976395 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:30.976924 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:31.977787 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:31.983525 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:31.983986 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:32.984802 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:32.989975 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:32.990401 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:33.991564 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:33.997677 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:33.998116 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:34.999040 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:35.004510 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:35.004931 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:36.032688 14467 auto_leader_rebalancer-test.cc:314] The leader distribution is
I20260810 18:50:36.033377 14467 tablet_server.cc:178] TabletServer@127.14.32.193:0 shutting down...
W20260810 18:50:36.040118 14777 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260810 18:50:36.051925 14777 consensus_peers.cc:487] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 -> Peer 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253): Couldn't send request to peer 49ef0f73074a4b6e9549f68ceb348ae5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:50:36.072901 14467 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:36.074206 14467 tablet_replica.cc:330] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5: stopping tablet replica
I20260810 18:50:36.075287 14467 raft_consensus.cc:2229] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 4 LEADER]: Raft consensus shutting down.
I20260810 18:50:36.076375 14467 raft_consensus.cc:2258] T 69e52bc7697d4473b77a1bd1f6df04f2 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:36.078907 14467 tablet_replica.cc:330] T f14b3e767f224ce2b789dfcfd0414b89 P 49ef0f73074a4b6e9549f68ceb348ae5: stopping tablet replica
I20260810 18:50:36.079528 14467 raft_consensus.cc:2229] T f14b3e767f224ce2b789dfcfd0414b89 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:36.079936 14467 raft_consensus.cc:2258] T f14b3e767f224ce2b789dfcfd0414b89 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:36.081611 14467 tablet_replica.cc:330] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5: stopping tablet replica
I20260810 18:50:36.082204 14467 raft_consensus.cc:2229] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 3 LEADER]: Raft consensus shutting down.
I20260810 18:50:36.083050 14467 raft_consensus.cc:2258] T a8187c40cb114a26b4da50c7bdbd4067 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:36.084818 14467 tablet_replica.cc:330] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5: stopping tablet replica
I20260810 18:50:36.085263 14467 raft_consensus.cc:2229] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:36.085692 14467 raft_consensus.cc:2258] T cd2a25bed1474a0692bcdfde37ab1d5b P 49ef0f73074a4b6e9549f68ceb348ae5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:36.087528 14467 tablet_replica.cc:330] T 9a2e5b7738d54ec99159df35d65b7b32 P 49ef0f73074a4b6e9549f68ceb348ae5: stopping tablet replica
I20260810 18:50:36.088034 14467 raft_consensus.cc:2229] T 9a2e5b7738d54ec99159df35d65b7b32 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:36.088442 14467 raft_consensus.cc:2258] T 9a2e5b7738d54ec99159df35d65b7b32 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:36.090672 14467 tablet_replica.cc:330] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5: stopping tablet replica
I20260810 18:50:36.091252 14467 raft_consensus.cc:2229] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [term 5 LEADER]: Raft consensus shutting down.
W20260810 18:50:36.092008 15375 consensus_queue.cc:1159] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "94af46e728ad452d91785ca419624f59" responder_term: 5 status { last_received { term: 5 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 5 index: 5 } }
I20260810 18:50:36.093137 14467 raft_consensus.cc:2258] T 395eacb81f614569895253d87e09b9ba P 49ef0f73074a4b6e9549f68ceb348ae5 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:36.095271 14467 tablet_replica.cc:330] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5: stopping tablet replica
I20260810 18:50:36.095858 14467 raft_consensus.cc:2229] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:36.096231 14467 raft_consensus.cc:2258] T 7977ab6121414743ab57535b534ea4f9 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:36.097831 14467 tablet_replica.cc:330] T 4ef3949841d54dfcb19e1f25582bbeaa P 49ef0f73074a4b6e9549f68ceb348ae5: stopping tablet replica
I20260810 18:50:36.098277 14467 raft_consensus.cc:2229] T 4ef3949841d54dfcb19e1f25582bbeaa P 49ef0f73074a4b6e9549f68ceb348ae5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:36.098678 14467 raft_consensus.cc:2258] T 4ef3949841d54dfcb19e1f25582bbeaa P 49ef0f73074a4b6e9549f68ceb348ae5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:36.100270 14467 tablet_replica.cc:330] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5: stopping tablet replica
I20260810 18:50:36.100730 14467 raft_consensus.cc:2229] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:36.101265 14467 raft_consensus.cc:2258] T 20ae3d07fdc14d4e9be86682e53a7d10 P 49ef0f73074a4b6e9549f68ceb348ae5 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260810 18:50:36.119952 14911 consensus_peers.cc:487] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 -> Peer 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253): Couldn't send request to peer 49ef0f73074a4b6e9549f68ceb348ae5. Status: Network error: Client connection negotiation failed: client connection to 127.14.32.193:45253: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260810 18:50:36.257511 14777 consensus_peers.cc:487] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59 -> Peer 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253): Couldn't send request to peer 49ef0f73074a4b6e9549f68ceb348ae5. Status: Network error: Client connection negotiation failed: client connection to 127.14.32.193:45253: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 18:50:36.275357 14467 tablet_server.cc:194] TabletServer@127.14.32.193:0 shutdown complete.
I20260810 18:50:36.293221 14467 tablet_server.cc:178] TabletServer@127.14.32.194:0 shutting down...
W20260810 18:50:36.323299 14911 consensus_peers.cc:487] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51 -> Peer 49ef0f73074a4b6e9549f68ceb348ae5 (127.14.32.193:45253): Couldn't send request to peer 49ef0f73074a4b6e9549f68ceb348ae5. Status: Network error: Client connection negotiation failed: client connection to 127.14.32.193:45253: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 18:50:36.325392 14467 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:36.326179 14467 tablet_replica.cc:330] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59: stopping tablet replica
I20260810 18:50:36.326799 14467 raft_consensus.cc:2229] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:36.327347 14467 raft_consensus.cc:2258] T 395eacb81f614569895253d87e09b9ba P 94af46e728ad452d91785ca419624f59 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:36.329119 14467 tablet_replica.cc:330] T 9a2e5b7738d54ec99159df35d65b7b32 P 94af46e728ad452d91785ca419624f59: stopping tablet replica
I20260810 18:50:36.329612 14467 raft_consensus.cc:2229] T 9a2e5b7738d54ec99159df35d65b7b32 P 94af46e728ad452d91785ca419624f59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:36.330016 14467 raft_consensus.cc:2258] T 9a2e5b7738d54ec99159df35d65b7b32 P 94af46e728ad452d91785ca419624f59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:36.332206 14467 tablet_replica.cc:330] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59: stopping tablet replica
I20260810 18:50:36.332765 14467 raft_consensus.cc:2229] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [term 3 LEADER]: Raft consensus shutting down.
I20260810 18:50:36.333796 14467 raft_consensus.cc:2258] T 7977ab6121414743ab57535b534ea4f9 P 94af46e728ad452d91785ca419624f59 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:36.335462 14467 tablet_replica.cc:330] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59: stopping tablet replica
I20260810 18:50:36.335953 14467 raft_consensus.cc:2229] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:36.336428 14467 raft_consensus.cc:2258] T a8187c40cb114a26b4da50c7bdbd4067 P 94af46e728ad452d91785ca419624f59 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:36.338088 14467 tablet_replica.cc:330] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59: stopping tablet replica
I20260810 18:50:36.338557 14467 raft_consensus.cc:2229] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:36.338970 14467 raft_consensus.cc:2258] T cd2a25bed1474a0692bcdfde37ab1d5b P 94af46e728ad452d91785ca419624f59 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:36.340633 14467 tablet_replica.cc:330] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59: stopping tablet replica
I20260810 18:50:36.341141 14467 raft_consensus.cc:2229] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:50:36.341956 14467 raft_consensus.cc:2258] T f14b3e767f224ce2b789dfcfd0414b89 P 94af46e728ad452d91785ca419624f59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:36.343580 14467 tablet_replica.cc:330] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59: stopping tablet replica
I20260810 18:50:36.344026 14467 raft_consensus.cc:2229] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:36.344413 14467 raft_consensus.cc:2258] T 4ef3949841d54dfcb19e1f25582bbeaa P 94af46e728ad452d91785ca419624f59 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:36.346009 14467 tablet_replica.cc:330] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59: stopping tablet replica
I20260810 18:50:36.346437 14467 raft_consensus.cc:2229] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59 [term 3 LEADER]: Raft consensus shutting down.
I20260810 18:50:36.347275 14467 raft_consensus.cc:2258] T 20ae3d07fdc14d4e9be86682e53a7d10 P 94af46e728ad452d91785ca419624f59 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:36.348774 14467 tablet_replica.cc:330] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59: stopping tablet replica
I20260810 18:50:36.349210 14467 raft_consensus.cc:2229] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:36.349589 14467 raft_consensus.cc:2258] T 69e52bc7697d4473b77a1bd1f6df04f2 P 94af46e728ad452d91785ca419624f59 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:36.377880 14467 tablet_server.cc:194] TabletServer@127.14.32.194:0 shutdown complete.
I20260810 18:50:36.395457 14467 tablet_server.cc:178] TabletServer@127.14.32.195:0 shutting down...
I20260810 18:50:36.429601 14467 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:36.430300 14467 tablet_replica.cc:330] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51: stopping tablet replica
I20260810 18:50:36.430979 14467 raft_consensus.cc:2229] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:36.431499 14467 raft_consensus.cc:2258] T 69e52bc7697d4473b77a1bd1f6df04f2 P b847dd1039674d29a3c811bdfb6c7f51 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:36.433238 14467 tablet_replica.cc:330] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51: stopping tablet replica
I20260810 18:50:36.433737 14467 raft_consensus.cc:2229] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:50:36.434551 14467 raft_consensus.cc:2258] T 9a2e5b7738d54ec99159df35d65b7b32 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:36.436321 14467 tablet_replica.cc:330] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51: stopping tablet replica
I20260810 18:50:36.436905 14467 raft_consensus.cc:2229] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:36.437315 14467 raft_consensus.cc:2258] T 7977ab6121414743ab57535b534ea4f9 P b847dd1039674d29a3c811bdfb6c7f51 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:36.438879 14467 tablet_replica.cc:330] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51: stopping tablet replica
I20260810 18:50:36.439428 14467 raft_consensus.cc:2229] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:36.439849 14467 raft_consensus.cc:2258] T 395eacb81f614569895253d87e09b9ba P b847dd1039674d29a3c811bdfb6c7f51 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:36.441601 14467 tablet_replica.cc:330] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51: stopping tablet replica
I20260810 18:50:36.442076 14467 raft_consensus.cc:2229] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [term 4 LEADER]: Raft consensus shutting down.
I20260810 18:50:36.442866 14467 raft_consensus.cc:2258] T cd2a25bed1474a0692bcdfde37ab1d5b P b847dd1039674d29a3c811bdfb6c7f51 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:36.444492 14467 tablet_replica.cc:330] T a8187c40cb114a26b4da50c7bdbd4067 P b847dd1039674d29a3c811bdfb6c7f51: stopping tablet replica
I20260810 18:50:36.444952 14467 raft_consensus.cc:2229] T a8187c40cb114a26b4da50c7bdbd4067 P b847dd1039674d29a3c811bdfb6c7f51 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:36.445353 14467 raft_consensus.cc:2258] T a8187c40cb114a26b4da50c7bdbd4067 P b847dd1039674d29a3c811bdfb6c7f51 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:36.446975 14467 tablet_replica.cc:330] T f14b3e767f224ce2b789dfcfd0414b89 P b847dd1039674d29a3c811bdfb6c7f51: stopping tablet replica
I20260810 18:50:36.447525 14467 raft_consensus.cc:2229] T f14b3e767f224ce2b789dfcfd0414b89 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:36.447896 14467 raft_consensus.cc:2258] T f14b3e767f224ce2b789dfcfd0414b89 P b847dd1039674d29a3c811bdfb6c7f51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:36.449491 14467 tablet_replica.cc:330] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51: stopping tablet replica
I20260810 18:50:36.449930 14467 raft_consensus.cc:2229] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [term 3 LEADER]: Raft consensus shutting down.
I20260810 18:50:36.450688 14467 raft_consensus.cc:2258] T 4ef3949841d54dfcb19e1f25582bbeaa P b847dd1039674d29a3c811bdfb6c7f51 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:36.452266 14467 tablet_replica.cc:330] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51: stopping tablet replica
I20260810 18:50:36.452696 14467 raft_consensus.cc:2229] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:36.453267 14467 raft_consensus.cc:2258] T 20ae3d07fdc14d4e9be86682e53a7d10 P b847dd1039674d29a3c811bdfb6c7f51 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:36.496906 14467 tablet_server.cc:194] TabletServer@127.14.32.195:0 shutdown complete.
I20260810 18:50:36.515969 14467 master.cc:537] Master@127.14.32.254:37379 shutting down...
I20260810 18:50:36.877604 14467 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7fe0a77d39a24aba944a1d2f29ce4eda [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:50:36.878252 14467 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7fe0a77d39a24aba944a1d2f29ce4eda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:36.878634 14467 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7fe0a77d39a24aba944a1d2f29ce4eda: stopping tablet replica
I20260810 18:50:36.915967 14467 master.cc:552] Master@127.14.32.254:37379 shutdown complete.
49ef0f73074a4b6e9549f68ceb348ae5 4
94af46e728ad452d91785ca419624f59 4
b847dd1039674d29a3c811bdfb6c7f51 1
49ef0f73074a4b6e9549f68ceb348ae5 3
94af46e728ad452d91785ca419624f59 3
b847dd1039674d29a3c811bdfb6c7f51 3
49ef0f73074a4b6e9549f68ceb348ae5 0
94af46e728ad452d91785ca419624f59 8
b847dd1039674d29a3c811bdfb6c7f51 1
49ef0f73074a4b6e9549f68ceb348ae5 3
94af46e728ad452d91785ca419624f59 3
b847dd1039674d29a3c811bdfb6c7f51 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (33490 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260810 18:50:36.958897 14467 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.14.32.254:34643
I20260810 18:50:36.960007 14467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:36.965642 15382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:36.965709 15381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:36.966873 15385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:36.968511 14467 server_base.cc:1034] running on GCE node
I20260810 18:50:36.969714 14467 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:36.969929 14467 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:36.970103 14467 hybrid_clock.cc:648] HybridClock initialized: now 1786387836970073 us; error 0 us; skew 500 ppm
I20260810 18:50:36.970639 14467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:36.973093 14467 webserver.cc:460] Webserver started at http://127.14.32.254:44475/ using document root <none> and password file <none>
I20260810 18:50:36.973589 14467 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:36.973773 14467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:36.974094 14467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:36.975770 14467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786387803181742-14467-0/minicluster-data/master-0-root/instance:
uuid: "78d70a72464749988488cc4d84aeb847"
format_stamp: "Formatted at 2026-08-10 18:50:36 on dist-test-slave-n326"
I20260810 18:50:36.980218 14467 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:50:36.983407 15392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:36.984095 14467 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:50:36.984376 14467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786387803181742-14467-0/minicluster-data/master-0-root
uuid: "78d70a72464749988488cc4d84aeb847"
format_stamp: "Formatted at 2026-08-10 18:50:36 on dist-test-slave-n326"
I20260810 18:50:36.984640 14467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786387803181742-14467-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786387803181742-14467-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786387803181742-14467-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:36.997145 14467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:36.998319 14467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:37.036089 14467 rpc_server.cc:307] RPC server started. Bound to: 127.14.32.254:34643
I20260810 18:50:37.036196 15490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.32.254:34643 every 8 connection(s)
I20260810 18:50:37.040345 15491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.049901 15491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d70a72464749988488cc4d84aeb847: Bootstrap starting.
I20260810 18:50:37.054378 15491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78d70a72464749988488cc4d84aeb847: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:37.058616 15491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d70a72464749988488cc4d84aeb847: No bootstrap required, opened a new log
I20260810 18:50:37.060601 15491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78d70a72464749988488cc4d84aeb847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d70a72464749988488cc4d84aeb847" member_type: VOTER }
I20260810 18:50:37.061022 15491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78d70a72464749988488cc4d84aeb847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:37.061290 15491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78d70a72464749988488cc4d84aeb847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d70a72464749988488cc4d84aeb847, State: Initialized, Role: FOLLOWER
I20260810 18:50:37.061901 15491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78d70a72464749988488cc4d84aeb847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d70a72464749988488cc4d84aeb847" member_type: VOTER }
I20260810 18:50:37.062340 15491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78d70a72464749988488cc4d84aeb847 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:50:37.062561 15491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78d70a72464749988488cc4d84aeb847 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:50:37.062888 15491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78d70a72464749988488cc4d84aeb847 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:37.066619 15491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78d70a72464749988488cc4d84aeb847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d70a72464749988488cc4d84aeb847" member_type: VOTER }
I20260810 18:50:37.067181 15491 leader_election.cc:302] T 00000000000000000000000000000000 P 78d70a72464749988488cc4d84aeb847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78d70a72464749988488cc4d84aeb847; no voters:
I20260810 18:50:37.068387 15491 leader_election.cc:288] T 00000000000000000000000000000000 P 78d70a72464749988488cc4d84aeb847 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:50:37.068840 15495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78d70a72464749988488cc4d84aeb847 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:37.070256 15495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78d70a72464749988488cc4d84aeb847 [term 1 LEADER]: Becoming Leader. State: Replica: 78d70a72464749988488cc4d84aeb847, State: Running, Role: LEADER
I20260810 18:50:37.070933 15495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78d70a72464749988488cc4d84aeb847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d70a72464749988488cc4d84aeb847" member_type: VOTER }
I20260810 18:50:37.071566 15491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78d70a72464749988488cc4d84aeb847 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:50:37.075969 15496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d70a72464749988488cc4d84aeb847 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78d70a72464749988488cc4d84aeb847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d70a72464749988488cc4d84aeb847" member_type: VOTER } }
I20260810 18:50:37.076529 15496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d70a72464749988488cc4d84aeb847 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:37.078297 15497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d70a72464749988488cc4d84aeb847 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d70a72464749988488cc4d84aeb847. Latest consensus state: current_term: 1 leader_uuid: "78d70a72464749988488cc4d84aeb847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d70a72464749988488cc4d84aeb847" member_type: VOTER } }
I20260810 18:50:37.078742 15497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d70a72464749988488cc4d84aeb847 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:37.082175 15507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:50:37.090346 15507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:50:37.091454 14467 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260810 18:50:37.098914 15507 catalog_manager.cc:1347] Generated new cluster ID: 0a40ca90c879497aa066237dc7468d59
I20260810 18:50:37.099242 15507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:50:37.110414 15507 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:50:37.111968 15507 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:50:37.125762 15507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78d70a72464749988488cc4d84aeb847: Generated new TSK 0
I20260810 18:50:37.126403 15507 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:50:37.158356 14467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:37.164316 15525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:37.165505 15526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:37.167768 15529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:37.168175 14467 server_base.cc:1034] running on GCE node
I20260810 18:50:37.169219 14467 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:37.169430 14467 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:37.169596 14467 hybrid_clock.cc:648] HybridClock initialized: now 1786387837169581 us; error 0 us; skew 500 ppm
I20260810 18:50:37.170120 14467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:37.172681 14467 webserver.cc:460] Webserver started at http://127.14.32.193:45745/ using document root <none> and password file <none>
I20260810 18:50:37.173230 14467 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:37.173453 14467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:37.173715 14467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:37.174793 14467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786387803181742-14467-0/minicluster-data/ts-0-root/instance:
uuid: "ffe600335b884cae8bdd12c2ef74e670"
format_stamp: "Formatted at 2026-08-10 18:50:37 on dist-test-slave-n326"
I20260810 18:50:37.179312 14467 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:50:37.182821 15535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:37.183614 14467 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:50:37.183941 14467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786387803181742-14467-0/minicluster-data/ts-0-root
uuid: "ffe600335b884cae8bdd12c2ef74e670"
format_stamp: "Formatted at 2026-08-10 18:50:37 on dist-test-slave-n326"
I20260810 18:50:37.184301 14467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786387803181742-14467-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786387803181742-14467-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786387803181742-14467-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:37.194018 14467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:37.195276 14467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:37.196713 14467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:37.198935 14467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:37.199230 14467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 18:50:37.199448 14467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:37.199586 14467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:37.242182 14467 rpc_server.cc:307] RPC server started. Bound to: 127.14.32.193:36539
I20260810 18:50:37.242277 15650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.32.193:36539 every 8 connection(s)
I20260810 18:50:37.247148 14467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:37.256767 15657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:37.259251 15659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:37.261013 14467 server_base.cc:1034] running on GCE node
W20260810 18:50:37.261188 15662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:37.262598 14467 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:37.262873 14467 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:37.263109 14467 hybrid_clock.cc:648] HybridClock initialized: now 1786387837263090 us; error 0 us; skew 500 ppm
I20260810 18:50:37.263768 14467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:37.266556 14467 webserver.cc:460] Webserver started at http://127.14.32.194:42439/ using document root <none> and password file <none>
I20260810 18:50:37.267205 14467 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:37.267417 14467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:37.267689 14467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:37.268853 14467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786387803181742-14467-0/minicluster-data/ts-1-root/instance:
uuid: "2d28250046da4d688d76be4ae6e4c826"
format_stamp: "Formatted at 2026-08-10 18:50:37 on dist-test-slave-n326"
I20260810 18:50:37.270130 15651 heartbeater.cc:347] Connected to a master server at 127.14.32.254:34643
I20260810 18:50:37.270484 15651 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:37.271221 15651 heartbeater.cc:511] Master 127.14.32.254:34643 requested a full tablet report, sending...
I20260810 18:50:37.273378 15419 ts_manager.cc:194] Registered new tserver with Master: ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539)
I20260810 18:50:37.275456 15419 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59738
I20260810 18:50:37.276015 14467 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 18:50:37.280166 15669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:37.280994 14467 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260810 18:50:37.281275 14467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786387803181742-14467-0/minicluster-data/ts-1-root
uuid: "2d28250046da4d688d76be4ae6e4c826"
format_stamp: "Formatted at 2026-08-10 18:50:37 on dist-test-slave-n326"
I20260810 18:50:37.281599 14467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786387803181742-14467-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786387803181742-14467-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786387803181742-14467-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:37.292119 14467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:37.293284 14467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:37.294697 14467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:37.296934 14467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:37.297153 14467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:37.297380 14467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:37.297549 14467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:37.337042 14467 rpc_server.cc:307] RPC server started. Bound to: 127.14.32.194:46629
I20260810 18:50:37.337154 15789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.32.194:46629 every 8 connection(s)
I20260810 18:50:37.341914 14467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:37.348719 15796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:37.349982 15800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:37.353262 14467 server_base.cc:1034] running on GCE node
W20260810 18:50:37.353952 15802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:37.354892 14467 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:37.355177 14467 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:37.355403 14467 hybrid_clock.cc:648] HybridClock initialized: now 1786387837355365 us; error 0 us; skew 500 ppm
I20260810 18:50:37.355616 15790 heartbeater.cc:347] Connected to a master server at 127.14.32.254:34643
I20260810 18:50:37.355975 15790 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:37.356108 14467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:37.356863 15790 heartbeater.cc:511] Master 127.14.32.254:34643 requested a full tablet report, sending...
I20260810 18:50:37.358781 14467 webserver.cc:460] Webserver started at http://127.14.32.195:38239/ using document root <none> and password file <none>
I20260810 18:50:37.359192 15419 ts_manager.cc:194] Registered new tserver with Master: 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629)
I20260810 18:50:37.359457 14467 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:37.359710 14467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:37.360010 14467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:37.361299 15419 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59744
I20260810 18:50:37.361610 14467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786387803181742-14467-0/minicluster-data/ts-2-root/instance:
uuid: "51eee958aa8948b1a46aa1641638b9ec"
format_stamp: "Formatted at 2026-08-10 18:50:37 on dist-test-slave-n326"
I20260810 18:50:37.366257 14467 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:50:37.369570 15814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:37.370350 14467 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:50:37.370682 14467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786387803181742-14467-0/minicluster-data/ts-2-root
uuid: "51eee958aa8948b1a46aa1641638b9ec"
format_stamp: "Formatted at 2026-08-10 18:50:37 on dist-test-slave-n326"
I20260810 18:50:37.370990 14467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786387803181742-14467-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786387803181742-14467-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786387803181742-14467-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:37.395637 14467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:37.396885 14467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:37.398281 14467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:37.400822 14467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:37.401094 14467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:37.401381 14467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:37.401574 14467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:37.442559 14467 rpc_server.cc:307] RPC server started. Bound to: 127.14.32.195:38891
I20260810 18:50:37.442662 15932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.32.195:38891 every 8 connection(s)
I20260810 18:50:37.457852 15933 heartbeater.cc:347] Connected to a master server at 127.14.32.254:34643
I20260810 18:50:37.458259 15933 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:37.458894 15933 heartbeater.cc:511] Master 127.14.32.254:34643 requested a full tablet report, sending...
I20260810 18:50:37.460775 15419 ts_manager.cc:194] Registered new tserver with Master: 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:37.461575 14467 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01572171s
I20260810 18:50:37.462335 15419 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59754
I20260810 18:50:37.463909 14467 test_util.cc:268] Using random seed: -730092212
I20260810 18:50:37.489341 15419 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59762:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:50:37.492226 15419 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:50:37.566241 15591 tablet_service.cc:1467] Processing CreateTablet for tablet 4f6b952757814de89b38a47f3942ee7e (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260810 18:50:37.567835 15591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f6b952757814de89b38a47f3942ee7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.568210 15588 tablet_service.cc:1467] Processing CreateTablet for tablet 20f91c7cf7b14e6ab5215e47458b4f1b (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260810 18:50:37.569080 15846 tablet_service.cc:1467] Processing CreateTablet for tablet 8e61e9bbff154604bce6712d94c73e90 (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260810 18:50:37.569536 15861 tablet_service.cc:1467] Processing CreateTablet for tablet 8b96127e94b74666aa7a5d280228ba7d (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260810 18:50:37.570394 15846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e61e9bbff154604bce6712d94c73e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.570919 15578 tablet_service.cc:1467] Processing CreateTablet for tablet 354460a175b24dcd873c370589a0f7d5 (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260810 18:50:37.573149 15577 tablet_service.cc:1467] Processing CreateTablet for tablet e7e5673440be4239a048129ffad7a53c (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260810 18:50:37.573937 15727 tablet_service.cc:1467] Processing CreateTablet for tablet 20f91c7cf7b14e6ab5215e47458b4f1b (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260810 18:50:37.575414 15857 tablet_service.cc:1467] Processing CreateTablet for tablet 4f6b952757814de89b38a47f3942ee7e (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260810 18:50:37.575898 15731 tablet_service.cc:1467] Processing CreateTablet for tablet 8b96127e94b74666aa7a5d280228ba7d (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260810 18:50:37.576239 15725 tablet_service.cc:1467] Processing CreateTablet for tablet 354460a175b24dcd873c370589a0f7d5 (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260810 18:50:37.576705 15857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f6b952757814de89b38a47f3942ee7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.578547 15721 tablet_service.cc:1467] Processing CreateTablet for tablet e7e5673440be4239a048129ffad7a53c (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260810 18:50:37.579720 15730 tablet_service.cc:1467] Processing CreateTablet for tablet 4f6b952757814de89b38a47f3942ee7e (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260810 18:50:37.580314 15573 tablet_service.cc:1467] Processing CreateTablet for tablet 2813a19d92094a648be20c4c20f713fa (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260810 18:50:37.580859 15716 tablet_service.cc:1467] Processing CreateTablet for tablet 2813a19d92094a648be20c4c20f713fa (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260810 18:50:37.581357 15861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b96127e94b74666aa7a5d280228ba7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.582597 15856 tablet_service.cc:1467] Processing CreateTablet for tablet 20f91c7cf7b14e6ab5215e47458b4f1b (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260810 18:50:37.583204 15715 tablet_service.cc:1467] Processing CreateTablet for tablet 8e61e9bbff154604bce6712d94c73e90 (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260810 18:50:37.583881 15856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20f91c7cf7b14e6ab5215e47458b4f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.585515 15711 tablet_service.cc:1467] Processing CreateTablet for tablet 9764b4e886a24c9db19a64075920a794 (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260810 18:50:37.585983 15572 tablet_service.cc:1467] Processing CreateTablet for tablet 8e61e9bbff154604bce6712d94c73e90 (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260810 18:50:37.588599 15727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20f91c7cf7b14e6ab5215e47458b4f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.588444 15855 tablet_service.cc:1467] Processing CreateTablet for tablet 354460a175b24dcd873c370589a0f7d5 (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260810 18:50:37.592447 15847 tablet_service.cc:1467] Processing CreateTablet for tablet 2813a19d92094a648be20c4c20f713fa (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260810 18:50:37.595439 15725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 354460a175b24dcd873c370589a0f7d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.595394 15844 tablet_service.cc:1467] Processing CreateTablet for tablet 9764b4e886a24c9db19a64075920a794 (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260810 18:50:37.600458 15571 tablet_service.cc:1467] Processing CreateTablet for tablet 9764b4e886a24c9db19a64075920a794 (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260810 18:50:37.602278 15707 tablet_service.cc:1467] Processing CreateTablet for tablet 6df3bb273c824e68b83d4f4afdcb5b66 (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260810 18:50:37.604620 15594 tablet_service.cc:1467] Processing CreateTablet for tablet 8b96127e94b74666aa7a5d280228ba7d (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260810 18:50:37.611418 15730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f6b952757814de89b38a47f3942ee7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.613063 15731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b96127e94b74666aa7a5d280228ba7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.613260 15850 tablet_service.cc:1467] Processing CreateTablet for tablet e7e5673440be4239a048129ffad7a53c (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260810 18:50:37.613719 15721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7e5673440be4239a048129ffad7a53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.569621 15588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20f91c7cf7b14e6ab5215e47458b4f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.615908 15843 tablet_service.cc:1467] Processing CreateTablet for tablet cd4103b9a0414848b8ad7e116bf4ded8 (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260810 18:50:37.618094 15844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9764b4e886a24c9db19a64075920a794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.618789 15843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd4103b9a0414848b8ad7e116bf4ded8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.597136 15709 tablet_service.cc:1467] Processing CreateTablet for tablet cd4103b9a0414848b8ad7e116bf4ded8 (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260810 18:50:37.621021 15711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9764b4e886a24c9db19a64075920a794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.623404 15716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2813a19d92094a648be20c4c20f713fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.625694 15850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7e5673440be4239a048129ffad7a53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.625806 15709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd4103b9a0414848b8ad7e116bf4ded8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.626094 15570 tablet_service.cc:1467] Processing CreateTablet for tablet cd4103b9a0414848b8ad7e116bf4ded8 (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260810 18:50:37.627121 15855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 354460a175b24dcd873c370589a0f7d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.629446 15715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e61e9bbff154604bce6712d94c73e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.630168 15567 tablet_service.cc:1467] Processing CreateTablet for tablet 6df3bb273c824e68b83d4f4afdcb5b66 (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260810 18:50:37.632053 15570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd4103b9a0414848b8ad7e116bf4ded8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.637800 15578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 354460a175b24dcd873c370589a0f7d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.637889 15707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6df3bb273c824e68b83d4f4afdcb5b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.639324 15594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b96127e94b74666aa7a5d280228ba7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.640733 15577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7e5673440be4239a048129ffad7a53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.642675 15573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2813a19d92094a648be20c4c20f713fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.649654 15959 tablet_bootstrap.cc:492] T 8e61e9bbff154604bce6712d94c73e90 P 51eee958aa8948b1a46aa1641638b9ec: Bootstrap starting.
I20260810 18:50:37.662837 15847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2813a19d92094a648be20c4c20f713fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.672298 15959 tablet_bootstrap.cc:654] T 8e61e9bbff154604bce6712d94c73e90 P 51eee958aa8948b1a46aa1641638b9ec: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:37.681504 15572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e61e9bbff154604bce6712d94c73e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.685139 15571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9764b4e886a24c9db19a64075920a794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.688084 15567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6df3bb273c824e68b83d4f4afdcb5b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.731858 15961 tablet_bootstrap.cc:492] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826: Bootstrap starting.
I20260810 18:50:37.737680 15959 tablet_bootstrap.cc:492] T 8e61e9bbff154604bce6712d94c73e90 P 51eee958aa8948b1a46aa1641638b9ec: No bootstrap required, opened a new log
I20260810 18:50:37.738255 15959 ts_tablet_manager.cc:1397] T 8e61e9bbff154604bce6712d94c73e90 P 51eee958aa8948b1a46aa1641638b9ec: Time spent bootstrapping tablet: real 0.089s user 0.009s sys 0.012s
I20260810 18:50:37.738871 15961 tablet_bootstrap.cc:654] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:37.740092 15844 tablet_service.cc:1467] Processing CreateTablet for tablet 6df3bb273c824e68b83d4f4afdcb5b66 (DEFAULT_TABLE table=test-workload [id=7576e8442fca4b3ba1504010f7b6a03c]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260810 18:50:37.741287 15844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6df3bb273c824e68b83d4f4afdcb5b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.759603 15959 raft_consensus.cc:348] T 8e61e9bbff154604bce6712d94c73e90 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:37.760282 15959 raft_consensus.cc:374] T 8e61e9bbff154604bce6712d94c73e90 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:37.760602 15959 raft_consensus.cc:729] T 8e61e9bbff154604bce6712d94c73e90 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51eee958aa8948b1a46aa1641638b9ec, State: Initialized, Role: FOLLOWER
I20260810 18:50:37.761312 15959 consensus_queue.cc:260] T 8e61e9bbff154604bce6712d94c73e90 P 51eee958aa8948b1a46aa1641638b9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:37.764761 15962 tablet_bootstrap.cc:492] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670: Bootstrap starting.
I20260810 18:50:37.793918 15933 heartbeater.cc:503] Master 127.14.32.254:34643 was elected leader, sending a full tablet report...
I20260810 18:50:37.794301 15968 raft_consensus.cc:482] T 8e61e9bbff154604bce6712d94c73e90 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:37.794724 15968 raft_consensus.cc:504] T 8e61e9bbff154604bce6712d94c73e90 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:37.799342 15959 ts_tablet_manager.cc:1428] T 8e61e9bbff154604bce6712d94c73e90 P 51eee958aa8948b1a46aa1641638b9ec: Time spent starting tablet: real 0.061s user 0.011s sys 0.016s
I20260810 18:50:37.800467 15959 tablet_bootstrap.cc:492] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec: Bootstrap starting.
I20260810 18:50:37.808143 15962 tablet_bootstrap.cc:654] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:37.810896 15753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e61e9bbff154604bce6712d94c73e90" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d28250046da4d688d76be4ae6e4c826" is_pre_election: true
I20260810 18:50:37.811132 15968 leader_election.cc:288] T 8e61e9bbff154604bce6712d94c73e90 P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629)
W20260810 18:50:37.813764 15816 leader_election.cc:341] T 8e61e9bbff154604bce6712d94c73e90 P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:37.815126 15959 tablet_bootstrap.cc:654] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:37.813831 15961 tablet_bootstrap.cc:492] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826: No bootstrap required, opened a new log
I20260810 18:50:37.820214 15961 ts_tablet_manager.cc:1397] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826: Time spent bootstrapping tablet: real 0.089s user 0.019s sys 0.037s
I20260810 18:50:37.822894 15961 raft_consensus.cc:348] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:37.823630 15961 raft_consensus.cc:374] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:37.823948 15961 raft_consensus.cc:729] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d28250046da4d688d76be4ae6e4c826, State: Initialized, Role: FOLLOWER
I20260810 18:50:37.824604 15961 consensus_queue.cc:260] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:37.834807 15962 tablet_bootstrap.cc:492] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670: No bootstrap required, opened a new log
I20260810 18:50:37.835326 15962 ts_tablet_manager.cc:1397] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670: Time spent bootstrapping tablet: real 0.071s user 0.017s sys 0.014s
I20260810 18:50:37.837976 15962 raft_consensus.cc:348] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:37.838685 15962 raft_consensus.cc:374] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:37.838973 15962 raft_consensus.cc:729] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Initialized, Role: FOLLOWER
I20260810 18:50:37.839820 15962 consensus_queue.cc:260] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:37.841086 15961 ts_tablet_manager.cc:1428] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826: Time spent starting tablet: real 0.021s user 0.011s sys 0.003s
W20260810 18:50:37.841169 15792 tablet.cc:2369] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:50:37.842954 15959 tablet_bootstrap.cc:492] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec: No bootstrap required, opened a new log
I20260810 18:50:37.843358 15959 ts_tablet_manager.cc:1397] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec: Time spent bootstrapping tablet: real 0.043s user 0.007s sys 0.006s
I20260810 18:50:37.844087 15651 heartbeater.cc:503] Master 127.14.32.254:34643 was elected leader, sending a full tablet report...
I20260810 18:50:37.844863 15613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e61e9bbff154604bce6712d94c73e90" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe600335b884cae8bdd12c2ef74e670" is_pre_election: true
I20260810 18:50:37.845045 15962 ts_tablet_manager.cc:1428] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s
I20260810 18:50:37.846158 15962 tablet_bootstrap.cc:492] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670: Bootstrap starting.
I20260810 18:50:37.845858 15959 raft_consensus.cc:348] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } }
I20260810 18:50:37.846583 15959 raft_consensus.cc:374] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260810 18:50:37.847128 15817 leader_election.cc:341] T 8e61e9bbff154604bce6712d94c73e90 P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:37.847409 15790 heartbeater.cc:503] Master 127.14.32.254:34643 was elected leader, sending a full tablet report...
I20260810 18:50:37.852101 15962 tablet_bootstrap.cc:654] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:37.852438 15817 leader_election.cc:302] T 8e61e9bbff154604bce6712d94c73e90 P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51eee958aa8948b1a46aa1641638b9ec; no voters: 2d28250046da4d688d76be4ae6e4c826, ffe600335b884cae8bdd12c2ef74e670
I20260810 18:50:37.846892 15959 raft_consensus.cc:729] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51eee958aa8948b1a46aa1641638b9ec, State: Initialized, Role: FOLLOWER
I20260810 18:50:37.848256 15961 tablet_bootstrap.cc:492] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826: Bootstrap starting.
I20260810 18:50:37.853264 15968 raft_consensus.cc:2726] T 8e61e9bbff154604bce6712d94c73e90 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:50:37.854458 15959 consensus_queue.cc:260] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } }
I20260810 18:50:37.858731 15959 ts_tablet_manager.cc:1428] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20260810 18:50:37.859473 15961 tablet_bootstrap.cc:654] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:37.859614 15959 tablet_bootstrap.cc:492] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec: Bootstrap starting.
I20260810 18:50:37.865912 15959 tablet_bootstrap.cc:654] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:37.866945 15961 tablet_bootstrap.cc:492] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826: No bootstrap required, opened a new log
I20260810 18:50:37.867429 15961 ts_tablet_manager.cc:1397] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.004s
I20260810 18:50:37.870537 15961 raft_consensus.cc:348] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:37.871098 15962 tablet_bootstrap.cc:492] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670: No bootstrap required, opened a new log
I20260810 18:50:37.871403 15961 raft_consensus.cc:374] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:37.871652 15962 ts_tablet_manager.cc:1397] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.004s
I20260810 18:50:37.871722 15961 raft_consensus.cc:729] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d28250046da4d688d76be4ae6e4c826, State: Initialized, Role: FOLLOWER
I20260810 18:50:37.872691 15959 tablet_bootstrap.cc:492] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec: No bootstrap required, opened a new log
I20260810 18:50:37.873162 15959 ts_tablet_manager.cc:1397] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.004s
I20260810 18:50:37.872778 15961 consensus_queue.cc:260] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:37.874657 15962 raft_consensus.cc:348] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:37.875293 15961 ts_tablet_manager.cc:1428] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260810 18:50:37.875448 15962 raft_consensus.cc:374] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:37.875805 15962 raft_consensus.cc:729] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Initialized, Role: FOLLOWER
I20260810 18:50:37.875698 15959 raft_consensus.cc:348] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:37.876247 15961 tablet_bootstrap.cc:492] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826: Bootstrap starting.
I20260810 18:50:37.876471 15959 raft_consensus.cc:374] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:37.876763 15959 raft_consensus.cc:729] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51eee958aa8948b1a46aa1641638b9ec, State: Initialized, Role: FOLLOWER
I20260810 18:50:37.877116 15977 raft_consensus.cc:482] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:37.876674 15962 consensus_queue.cc:260] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:37.877537 15977 raft_consensus.cc:504] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:37.877559 15959 consensus_queue.cc:260] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:37.879199 15962 ts_tablet_manager.cc:1428] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 18:50:37.879805 15959 ts_tablet_manager.cc:1428] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260810 18:50:37.880074 15962 tablet_bootstrap.cc:492] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670: Bootstrap starting.
I20260810 18:50:37.880779 15959 tablet_bootstrap.cc:492] T 20f91c7cf7b14e6ab5215e47458b4f1b P 51eee958aa8948b1a46aa1641638b9ec: Bootstrap starting.
I20260810 18:50:37.886298 15962 tablet_bootstrap.cc:654] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:37.887149 15959 tablet_bootstrap.cc:654] T 20f91c7cf7b14e6ab5215e47458b4f1b P 51eee958aa8948b1a46aa1641638b9ec: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:37.896708 15977 leader_election.cc:288] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:37.901777 15962 tablet_bootstrap.cc:492] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670: No bootstrap required, opened a new log
I20260810 18:50:37.901954 15959 tablet_bootstrap.cc:492] T 20f91c7cf7b14e6ab5215e47458b4f1b P 51eee958aa8948b1a46aa1641638b9ec: No bootstrap required, opened a new log
I20260810 18:50:37.902263 15962 ts_tablet_manager.cc:1397] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.004s
I20260810 18:50:37.902449 15959 ts_tablet_manager.cc:1397] T 20f91c7cf7b14e6ab5215e47458b4f1b P 51eee958aa8948b1a46aa1641638b9ec: Time spent bootstrapping tablet: real 0.022s user 0.015s sys 0.003s
I20260810 18:50:37.904948 15962 raft_consensus.cc:348] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:37.905211 15959 raft_consensus.cc:348] T 20f91c7cf7b14e6ab5215e47458b4f1b P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:37.905710 15962 raft_consensus.cc:374] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:37.905970 15959 raft_consensus.cc:374] T 20f91c7cf7b14e6ab5215e47458b4f1b P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:37.906028 15962 raft_consensus.cc:729] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Initialized, Role: FOLLOWER
I20260810 18:50:37.906361 15959 raft_consensus.cc:729] T 20f91c7cf7b14e6ab5215e47458b4f1b P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51eee958aa8948b1a46aa1641638b9ec, State: Initialized, Role: FOLLOWER
I20260810 18:50:37.907009 15962 consensus_queue.cc:260] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:37.907157 15959 consensus_queue.cc:260] T 20f91c7cf7b14e6ab5215e47458b4f1b P 51eee958aa8948b1a46aa1641638b9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:37.909693 15959 ts_tablet_manager.cc:1428] T 20f91c7cf7b14e6ab5215e47458b4f1b P 51eee958aa8948b1a46aa1641638b9ec: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260810 18:50:37.910597 15959 tablet_bootstrap.cc:492] T cd4103b9a0414848b8ad7e116bf4ded8 P 51eee958aa8948b1a46aa1641638b9ec: Bootstrap starting.
I20260810 18:50:37.916736 15959 tablet_bootstrap.cc:654] T cd4103b9a0414848b8ad7e116bf4ded8 P 51eee958aa8948b1a46aa1641638b9ec: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:37.917518 15962 ts_tablet_manager.cc:1428] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20260810 18:50:37.921517 15613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2813a19d92094a648be20c4c20f713fa" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe600335b884cae8bdd12c2ef74e670" is_pre_election: true
I20260810 18:50:37.922364 15613 raft_consensus.cc:2454] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d28250046da4d688d76be4ae6e4c826 in term 0.
I20260810 18:50:37.923477 15968 raft_consensus.cc:482] T 20f91c7cf7b14e6ab5215e47458b4f1b P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:37.924052 15968 raft_consensus.cc:504] T 20f91c7cf7b14e6ab5215e47458b4f1b P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:37.926244 15968 leader_election.cc:288] T 20f91c7cf7b14e6ab5215e47458b4f1b P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629)
I20260810 18:50:37.928035 15962 tablet_bootstrap.cc:492] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670: Bootstrap starting.
I20260810 18:50:37.928556 15753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f91c7cf7b14e6ab5215e47458b4f1b" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d28250046da4d688d76be4ae6e4c826" is_pre_election: true
I20260810 18:50:37.928670 15613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f91c7cf7b14e6ab5215e47458b4f1b" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe600335b884cae8bdd12c2ef74e670" is_pre_election: true
W20260810 18:50:37.929821 15816 leader_election.cc:341] T 20f91c7cf7b14e6ab5215e47458b4f1b P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:37.930027 15979 raft_consensus.cc:482] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:37.930835 15961 tablet_bootstrap.cc:654] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:37.930503 15979 raft_consensus.cc:504] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:37.930766 15879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2813a19d92094a648be20c4c20f713fa" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51eee958aa8948b1a46aa1641638b9ec" is_pre_election: true
I20260810 18:50:37.930797 15675 leader_election.cc:302] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d28250046da4d688d76be4ae6e4c826, ffe600335b884cae8bdd12c2ef74e670; no voters:
I20260810 18:50:37.931893 15977 raft_consensus.cc:2781] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:37.932349 15977 raft_consensus.cc:482] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:37.932746 15977 raft_consensus.cc:3037] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:37.933080 15979 leader_election.cc:288] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
W20260810 18:50:37.935437 15673 leader_election.cc:341] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891): Illegal state: must be running to vote when last-logged opid is not known
W20260810 18:50:37.936058 15817 leader_election.cc:341] T 20f91c7cf7b14e6ab5215e47458b4f1b P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:37.936555 15817 leader_election.cc:302] T 20f91c7cf7b14e6ab5215e47458b4f1b P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51eee958aa8948b1a46aa1641638b9ec; no voters: 2d28250046da4d688d76be4ae6e4c826, ffe600335b884cae8bdd12c2ef74e670
I20260810 18:50:37.943121 15968 raft_consensus.cc:2726] T 20f91c7cf7b14e6ab5215e47458b4f1b P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:50:37.951608 15959 tablet_bootstrap.cc:492] T cd4103b9a0414848b8ad7e116bf4ded8 P 51eee958aa8948b1a46aa1641638b9ec: No bootstrap required, opened a new log
I20260810 18:50:37.952726 15959 ts_tablet_manager.cc:1397] T cd4103b9a0414848b8ad7e116bf4ded8 P 51eee958aa8948b1a46aa1641638b9ec: Time spent bootstrapping tablet: real 0.042s user 0.018s sys 0.008s
I20260810 18:50:37.953657 15962 tablet_bootstrap.cc:654] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:37.949148 15977 raft_consensus.cc:504] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:37.955991 15961 tablet_bootstrap.cc:492] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826: No bootstrap required, opened a new log
I20260810 18:50:37.956499 15961 ts_tablet_manager.cc:1397] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826: Time spent bootstrapping tablet: real 0.080s user 0.013s sys 0.005s
I20260810 18:50:37.957674 15613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2813a19d92094a648be20c4c20f713fa" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
I20260810 18:50:37.958344 15613 raft_consensus.cc:3037] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:37.958827 15959 raft_consensus.cc:348] T cd4103b9a0414848b8ad7e116bf4ded8 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:37.959424 15879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2813a19d92094a648be20c4c20f713fa" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
I20260810 18:50:37.959606 15959 raft_consensus.cc:374] T cd4103b9a0414848b8ad7e116bf4ded8 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:37.959932 15959 raft_consensus.cc:729] T cd4103b9a0414848b8ad7e116bf4ded8 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51eee958aa8948b1a46aa1641638b9ec, State: Initialized, Role: FOLLOWER
W20260810 18:50:37.960887 15673 leader_election.cc:341] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:37.960618 15959 consensus_queue.cc:260] T cd4103b9a0414848b8ad7e116bf4ded8 P 51eee958aa8948b1a46aa1641638b9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:37.958294 15977 leader_election.cc:288] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 1 election: Requested vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:37.966212 15961 raft_consensus.cc:348] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:37.966981 15961 raft_consensus.cc:374] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:37.967315 15961 raft_consensus.cc:729] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d28250046da4d688d76be4ae6e4c826, State: Initialized, Role: FOLLOWER
I20260810 18:50:37.969991 15959 ts_tablet_manager.cc:1428] T cd4103b9a0414848b8ad7e116bf4ded8 P 51eee958aa8948b1a46aa1641638b9ec: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260810 18:50:37.970278 15613 raft_consensus.cc:2454] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d28250046da4d688d76be4ae6e4c826 in term 1.
I20260810 18:50:37.971841 15675 leader_election.cc:302] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d28250046da4d688d76be4ae6e4c826, ffe600335b884cae8bdd12c2ef74e670; no voters: 51eee958aa8948b1a46aa1641638b9ec
I20260810 18:50:37.972842 15977 raft_consensus.cc:2781] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:37.973208 15961 consensus_queue.cc:260] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:37.974478 15959 tablet_bootstrap.cc:492] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec: Bootstrap starting.
I20260810 18:50:37.979272 15977 raft_consensus.cc:686] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [term 1 LEADER]: Becoming Leader. State: Replica: 2d28250046da4d688d76be4ae6e4c826, State: Running, Role: LEADER
I20260810 18:50:37.983381 15962 tablet_bootstrap.cc:492] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670: No bootstrap required, opened a new log
I20260810 18:50:37.983871 15962 ts_tablet_manager.cc:1397] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670: Time spent bootstrapping tablet: real 0.064s user 0.018s sys 0.005s
I20260810 18:50:37.984159 15753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9764b4e886a24c9db19a64075920a794" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d28250046da4d688d76be4ae6e4c826" is_pre_election: true
I20260810 18:50:37.984833 15753 raft_consensus.cc:2454] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffe600335b884cae8bdd12c2ef74e670 in term 0.
I20260810 18:50:37.986301 15537 leader_election.cc:302] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d28250046da4d688d76be4ae6e4c826, ffe600335b884cae8bdd12c2ef74e670; no voters:
I20260810 18:50:37.986666 15879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9764b4e886a24c9db19a64075920a794" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51eee958aa8948b1a46aa1641638b9ec" is_pre_election: true
I20260810 18:50:37.987242 15979 raft_consensus.cc:2781] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:37.987633 15979 raft_consensus.cc:482] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:37.987403 15962 raft_consensus.cc:348] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:37.987985 15979 raft_consensus.cc:3037] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:37.988276 15962 raft_consensus.cc:374] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:37.984308 15977 consensus_queue.cc:237] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:37.988610 15962 raft_consensus.cc:729] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Initialized, Role: FOLLOWER
I20260810 18:50:37.989524 15962 consensus_queue.cc:260] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
W20260810 18:50:37.991180 15537 leader_election.cc:341] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:37.993979 15961 ts_tablet_manager.cc:1428] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826: Time spent starting tablet: real 0.037s user 0.010s sys 0.004s
I20260810 18:50:37.995290 15961 tablet_bootstrap.cc:492] T 8b96127e94b74666aa7a5d280228ba7d P 2d28250046da4d688d76be4ae6e4c826: Bootstrap starting.
I20260810 18:50:37.996309 15979 raft_consensus.cc:504] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:37.999408 15959 tablet_bootstrap.cc:654] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:37.999560 15753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9764b4e886a24c9db19a64075920a794" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
I20260810 18:50:38.000346 15753 raft_consensus.cc:3037] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:37.998865 15979 leader_election.cc:288] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 1 election: Requested vote from peers 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:38.008665 15879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9764b4e886a24c9db19a64075920a794" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
W20260810 18:50:38.010465 15537 leader_election.cc:341] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:38.014629 15753 raft_consensus.cc:2454] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffe600335b884cae8bdd12c2ef74e670 in term 1.
I20260810 18:50:38.017709 15959 tablet_bootstrap.cc:492] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec: No bootstrap required, opened a new log
I20260810 18:50:38.019048 15959 ts_tablet_manager.cc:1397] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.008s
I20260810 18:50:38.019887 15962 ts_tablet_manager.cc:1428] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670: Time spent starting tablet: real 0.036s user 0.019s sys 0.012s
I20260810 18:50:38.020737 15961 tablet_bootstrap.cc:654] T 8b96127e94b74666aa7a5d280228ba7d P 2d28250046da4d688d76be4ae6e4c826: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:38.020362 15417 catalog_manager.cc:5495] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194). New cstate: current_term: 1 leader_uuid: "2d28250046da4d688d76be4ae6e4c826" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:38.021780 15959 raft_consensus.cc:348] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.022487 15959 raft_consensus.cc:374] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:38.022768 15959 raft_consensus.cc:729] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51eee958aa8948b1a46aa1641638b9ec, State: Initialized, Role: FOLLOWER
I20260810 18:50:38.023643 15962 tablet_bootstrap.cc:492] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670: Bootstrap starting.
I20260810 18:50:38.023923 15959 consensus_queue.cc:260] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.026433 15959 ts_tablet_manager.cc:1428] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260810 18:50:38.027139 15537 leader_election.cc:302] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d28250046da4d688d76be4ae6e4c826, ffe600335b884cae8bdd12c2ef74e670; no voters: 51eee958aa8948b1a46aa1641638b9ec
I20260810 18:50:38.028486 16006 raft_consensus.cc:2781] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:38.029009 15959 tablet_bootstrap.cc:492] T 9764b4e886a24c9db19a64075920a794 P 51eee958aa8948b1a46aa1641638b9ec: Bootstrap starting.
I20260810 18:50:38.029160 16006 raft_consensus.cc:686] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [term 1 LEADER]: Becoming Leader. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Running, Role: LEADER
I20260810 18:50:38.030000 16006 consensus_queue.cc:237] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.031610 15962 tablet_bootstrap.cc:654] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:38.035681 15959 tablet_bootstrap.cc:654] T 9764b4e886a24c9db19a64075920a794 P 51eee958aa8948b1a46aa1641638b9ec: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:38.049597 15977 raft_consensus.cc:482] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:38.050287 15977 raft_consensus.cc:504] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.053381 15977 leader_election.cc:288] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:38.053870 15613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f6b952757814de89b38a47f3942ee7e" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe600335b884cae8bdd12c2ef74e670" is_pre_election: true
I20260810 18:50:38.056071 15879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f6b952757814de89b38a47f3942ee7e" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51eee958aa8948b1a46aa1641638b9ec" is_pre_election: true
I20260810 18:50:38.056914 15879 raft_consensus.cc:2454] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d28250046da4d688d76be4ae6e4c826 in term 0.
I20260810 18:50:38.054764 15613 raft_consensus.cc:2454] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d28250046da4d688d76be4ae6e4c826 in term 0.
I20260810 18:50:38.058915 15675 leader_election.cc:302] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d28250046da4d688d76be4ae6e4c826, ffe600335b884cae8bdd12c2ef74e670; no voters:
I20260810 18:50:38.060009 15977 raft_consensus.cc:2781] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:38.060452 15977 raft_consensus.cc:482] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:38.060860 15977 raft_consensus.cc:3037] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:38.076215 15977 raft_consensus.cc:504] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.080063 15613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f6b952757814de89b38a47f3942ee7e" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
I20260810 18:50:38.080787 15613 raft_consensus.cc:3037] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Advancing to term 1
W20260810 18:50:38.082161 15509 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260810 18:50:38.088829 15613 raft_consensus.cc:2454] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d28250046da4d688d76be4ae6e4c826 in term 1.
I20260810 18:50:38.090137 15675 leader_election.cc:302] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d28250046da4d688d76be4ae6e4c826, ffe600335b884cae8bdd12c2ef74e670; no voters:
I20260810 18:50:38.091446 16004 raft_consensus.cc:2781] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:38.092096 16004 raft_consensus.cc:686] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [term 1 LEADER]: Becoming Leader. State: Replica: 2d28250046da4d688d76be4ae6e4c826, State: Running, Role: LEADER
I20260810 18:50:38.093142 16004 consensus_queue.cc:237] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.101833 15879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f6b952757814de89b38a47f3942ee7e" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
I20260810 18:50:38.102649 15879 raft_consensus.cc:3037] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:38.111892 15977 leader_election.cc:288] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 1 election: Requested vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:38.117401 15879 raft_consensus.cc:2454] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d28250046da4d688d76be4ae6e4c826 in term 1.
I20260810 18:50:38.119557 15961 tablet_bootstrap.cc:492] T 8b96127e94b74666aa7a5d280228ba7d P 2d28250046da4d688d76be4ae6e4c826: No bootstrap required, opened a new log
I20260810 18:50:38.120184 15961 ts_tablet_manager.cc:1397] T 8b96127e94b74666aa7a5d280228ba7d P 2d28250046da4d688d76be4ae6e4c826: Time spent bootstrapping tablet: real 0.125s user 0.033s sys 0.018s
I20260810 18:50:38.124739 15959 tablet_bootstrap.cc:492] T 9764b4e886a24c9db19a64075920a794 P 51eee958aa8948b1a46aa1641638b9ec: No bootstrap required, opened a new log
I20260810 18:50:38.125348 15959 ts_tablet_manager.cc:1397] T 9764b4e886a24c9db19a64075920a794 P 51eee958aa8948b1a46aa1641638b9ec: Time spent bootstrapping tablet: real 0.097s user 0.013s sys 0.032s
I20260810 18:50:38.127748 15961 raft_consensus.cc:348] T 8b96127e94b74666aa7a5d280228ba7d P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } }
I20260810 18:50:38.128474 15961 raft_consensus.cc:374] T 8b96127e94b74666aa7a5d280228ba7d P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:38.128726 15959 raft_consensus.cc:348] T 9764b4e886a24c9db19a64075920a794 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.129786 15959 raft_consensus.cc:374] T 9764b4e886a24c9db19a64075920a794 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:38.130144 15959 raft_consensus.cc:729] T 9764b4e886a24c9db19a64075920a794 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51eee958aa8948b1a46aa1641638b9ec, State: Initialized, Role: FOLLOWER
I20260810 18:50:38.131397 15959 consensus_queue.cc:260] T 9764b4e886a24c9db19a64075920a794 P 51eee958aa8948b1a46aa1641638b9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.130990 15417 catalog_manager.cc:5495] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194). New cstate: current_term: 1 leader_uuid: "2d28250046da4d688d76be4ae6e4c826" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:38.128742 15961 raft_consensus.cc:729] T 8b96127e94b74666aa7a5d280228ba7d P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d28250046da4d688d76be4ae6e4c826, State: Initialized, Role: FOLLOWER
I20260810 18:50:38.135998 15961 consensus_queue.cc:260] T 8b96127e94b74666aa7a5d280228ba7d P 2d28250046da4d688d76be4ae6e4c826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } }
I20260810 18:50:38.138378 15961 ts_tablet_manager.cc:1428] T 8b96127e94b74666aa7a5d280228ba7d P 2d28250046da4d688d76be4ae6e4c826: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20260810 18:50:38.139351 15961 tablet_bootstrap.cc:492] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826: Bootstrap starting.
I20260810 18:50:38.140782 15959 ts_tablet_manager.cc:1428] T 9764b4e886a24c9db19a64075920a794 P 51eee958aa8948b1a46aa1641638b9ec: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20260810 18:50:38.145200 16006 raft_consensus.cc:482] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:38.146538 15961 tablet_bootstrap.cc:654] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:38.146415 16006 raft_consensus.cc:504] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.145866 15418 catalog_manager.cc:5495] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 reported cstate change: term changed from 0 to 1, leader changed from <none> to ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193). New cstate: current_term: 1 leader_uuid: "ffe600335b884cae8bdd12c2ef74e670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:38.149649 16006 leader_election.cc:288] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:38.145897 15959 tablet_bootstrap.cc:492] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec: Bootstrap starting.
I20260810 18:50:38.149751 15753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e61e9bbff154604bce6712d94c73e90" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d28250046da4d688d76be4ae6e4c826" is_pre_election: true
I20260810 18:50:38.150295 15879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e61e9bbff154604bce6712d94c73e90" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51eee958aa8948b1a46aa1641638b9ec" is_pre_election: true
I20260810 18:50:38.150894 15879 raft_consensus.cc:2454] T 8e61e9bbff154604bce6712d94c73e90 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffe600335b884cae8bdd12c2ef74e670 in term 0.
W20260810 18:50:38.151688 15537 leader_election.cc:341] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:38.152611 15537 leader_election.cc:302] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51eee958aa8948b1a46aa1641638b9ec, ffe600335b884cae8bdd12c2ef74e670; no voters: 2d28250046da4d688d76be4ae6e4c826
I20260810 18:50:38.154078 16006 raft_consensus.cc:2781] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:38.154531 16006 raft_consensus.cc:482] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:38.154910 16006 raft_consensus.cc:3037] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:38.158255 15959 tablet_bootstrap.cc:654] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:38.160254 15962 tablet_bootstrap.cc:492] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670: No bootstrap required, opened a new log
I20260810 18:50:38.160426 15961 tablet_bootstrap.cc:492] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826: No bootstrap required, opened a new log
I20260810 18:50:38.160739 15962 ts_tablet_manager.cc:1397] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670: Time spent bootstrapping tablet: real 0.137s user 0.065s sys 0.054s
I20260810 18:50:38.160892 15961 ts_tablet_manager.cc:1397] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826: Time spent bootstrapping tablet: real 0.022s user 0.016s sys 0.000s
I20260810 18:50:38.163445 15962 raft_consensus.cc:348] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.164192 15962 raft_consensus.cc:374] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:38.164506 15962 raft_consensus.cc:729] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Initialized, Role: FOLLOWER
I20260810 18:50:38.164314 15961 raft_consensus.cc:348] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.165333 15961 raft_consensus.cc:374] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:38.165704 15961 raft_consensus.cc:729] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d28250046da4d688d76be4ae6e4c826, State: Initialized, Role: FOLLOWER
I20260810 18:50:38.166765 15961 consensus_queue.cc:260] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.169507 15961 ts_tablet_manager.cc:1428] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260810 18:50:38.170110 15959 tablet_bootstrap.cc:492] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec: No bootstrap required, opened a new log
I20260810 18:50:38.170416 15961 tablet_bootstrap.cc:492] T 6df3bb273c824e68b83d4f4afdcb5b66 P 2d28250046da4d688d76be4ae6e4c826: Bootstrap starting.
I20260810 18:50:38.170558 15959 ts_tablet_manager.cc:1397] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.008s
I20260810 18:50:38.173177 15962 consensus_queue.cc:260] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.173290 15959 raft_consensus.cc:348] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.174260 15959 raft_consensus.cc:374] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:38.174531 15959 raft_consensus.cc:729] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51eee958aa8948b1a46aa1641638b9ec, State: Initialized, Role: FOLLOWER
I20260810 18:50:38.176561 15962 ts_tablet_manager.cc:1428] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260810 18:50:38.177039 16006 raft_consensus.cc:504] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.177924 15959 consensus_queue.cc:260] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.179955 15959 ts_tablet_manager.cc:1428] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260810 18:50:38.180074 15753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e61e9bbff154604bce6712d94c73e90" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
I20260810 18:50:38.181125 15959 tablet_bootstrap.cc:492] T 2813a19d92094a648be20c4c20f713fa P 51eee958aa8948b1a46aa1641638b9ec: Bootstrap starting.
I20260810 18:50:38.181676 15879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e61e9bbff154604bce6712d94c73e90" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
I20260810 18:50:38.182186 15879 raft_consensus.cc:3037] T 8e61e9bbff154604bce6712d94c73e90 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Advancing to term 1
W20260810 18:50:38.186352 15537 leader_election.cc:341] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:38.186591 15959 tablet_bootstrap.cc:654] T 2813a19d92094a648be20c4c20f713fa P 51eee958aa8948b1a46aa1641638b9ec: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:38.177531 15962 tablet_bootstrap.cc:492] T 6df3bb273c824e68b83d4f4afdcb5b66 P ffe600335b884cae8bdd12c2ef74e670: Bootstrap starting.
I20260810 18:50:38.188521 15961 tablet_bootstrap.cc:654] T 6df3bb273c824e68b83d4f4afdcb5b66 P 2d28250046da4d688d76be4ae6e4c826: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:38.194418 15962 tablet_bootstrap.cc:654] T 6df3bb273c824e68b83d4f4afdcb5b66 P ffe600335b884cae8bdd12c2ef74e670: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:38.198709 15879 raft_consensus.cc:2454] T 8e61e9bbff154604bce6712d94c73e90 P 51eee958aa8948b1a46aa1641638b9ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffe600335b884cae8bdd12c2ef74e670 in term 1.
I20260810 18:50:38.200322 15961 tablet_bootstrap.cc:492] T 6df3bb273c824e68b83d4f4afdcb5b66 P 2d28250046da4d688d76be4ae6e4c826: No bootstrap required, opened a new log
I20260810 18:50:38.201161 15959 tablet_bootstrap.cc:492] T 2813a19d92094a648be20c4c20f713fa P 51eee958aa8948b1a46aa1641638b9ec: No bootstrap required, opened a new log
I20260810 18:50:38.202513 15961 ts_tablet_manager.cc:1397] T 6df3bb273c824e68b83d4f4afdcb5b66 P 2d28250046da4d688d76be4ae6e4c826: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.003s
I20260810 18:50:38.202755 15959 ts_tablet_manager.cc:1397] T 2813a19d92094a648be20c4c20f713fa P 51eee958aa8948b1a46aa1641638b9ec: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.008s
I20260810 18:50:38.204241 15962 tablet_bootstrap.cc:492] T 6df3bb273c824e68b83d4f4afdcb5b66 P ffe600335b884cae8bdd12c2ef74e670: No bootstrap required, opened a new log
I20260810 18:50:38.201717 16006 leader_election.cc:288] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 1 election: Requested vote from peers 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:38.204676 15962 ts_tablet_manager.cc:1397] T 6df3bb273c824e68b83d4f4afdcb5b66 P ffe600335b884cae8bdd12c2ef74e670: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.001s
I20260810 18:50:38.205696 15961 raft_consensus.cc:348] T 6df3bb273c824e68b83d4f4afdcb5b66 P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.206425 15961 raft_consensus.cc:374] T 6df3bb273c824e68b83d4f4afdcb5b66 P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:38.206166 15959 raft_consensus.cc:348] T 2813a19d92094a648be20c4c20f713fa P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.206738 15961 raft_consensus.cc:729] T 6df3bb273c824e68b83d4f4afdcb5b66 P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d28250046da4d688d76be4ae6e4c826, State: Initialized, Role: FOLLOWER
I20260810 18:50:38.207017 15537 leader_election.cc:302] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51eee958aa8948b1a46aa1641638b9ec, ffe600335b884cae8bdd12c2ef74e670; no voters: 2d28250046da4d688d76be4ae6e4c826
I20260810 18:50:38.207134 15959 raft_consensus.cc:374] T 2813a19d92094a648be20c4c20f713fa P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:38.207710 15959 raft_consensus.cc:729] T 2813a19d92094a648be20c4c20f713fa P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51eee958aa8948b1a46aa1641638b9ec, State: Initialized, Role: FOLLOWER
I20260810 18:50:38.207350 15962 raft_consensus.cc:348] T 6df3bb273c824e68b83d4f4afdcb5b66 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.207499 15961 consensus_queue.cc:260] T 6df3bb273c824e68b83d4f4afdcb5b66 P 2d28250046da4d688d76be4ae6e4c826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.208223 16006 raft_consensus.cc:2781] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:38.208331 15962 raft_consensus.cc:374] T 6df3bb273c824e68b83d4f4afdcb5b66 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:38.208794 15962 raft_consensus.cc:729] T 6df3bb273c824e68b83d4f4afdcb5b66 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Initialized, Role: FOLLOWER
I20260810 18:50:38.208633 15959 consensus_queue.cc:260] T 2813a19d92094a648be20c4c20f713fa P 51eee958aa8948b1a46aa1641638b9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.209519 15962 consensus_queue.cc:260] T 6df3bb273c824e68b83d4f4afdcb5b66 P ffe600335b884cae8bdd12c2ef74e670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.210484 15961 ts_tablet_manager.cc:1428] T 6df3bb273c824e68b83d4f4afdcb5b66 P 2d28250046da4d688d76be4ae6e4c826: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260810 18:50:38.211417 15961 tablet_bootstrap.cc:492] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826: Bootstrap starting.
I20260810 18:50:38.211746 15962 ts_tablet_manager.cc:1428] T 6df3bb273c824e68b83d4f4afdcb5b66 P ffe600335b884cae8bdd12c2ef74e670: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20260810 18:50:38.212751 15962 tablet_bootstrap.cc:492] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670: Bootstrap starting.
I20260810 18:50:38.213912 15959 ts_tablet_manager.cc:1428] T 2813a19d92094a648be20c4c20f713fa P 51eee958aa8948b1a46aa1641638b9ec: Time spent starting tablet: real 0.011s user 0.004s sys 0.001s
I20260810 18:50:38.214838 15959 tablet_bootstrap.cc:492] T 6df3bb273c824e68b83d4f4afdcb5b66 P 51eee958aa8948b1a46aa1641638b9ec: Bootstrap starting.
I20260810 18:50:38.218685 15962 tablet_bootstrap.cc:654] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:38.219914 15961 tablet_bootstrap.cc:654] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:38.220079 15959 tablet_bootstrap.cc:654] T 6df3bb273c824e68b83d4f4afdcb5b66 P 51eee958aa8948b1a46aa1641638b9ec: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:38.229877 15962 tablet_bootstrap.cc:492] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670: No bootstrap required, opened a new log
I20260810 18:50:38.230396 15962 ts_tablet_manager.cc:1397] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.008s
I20260810 18:50:38.231437 16006 raft_consensus.cc:686] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [term 1 LEADER]: Becoming Leader. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Running, Role: LEADER
I20260810 18:50:38.232321 15961 tablet_bootstrap.cc:492] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826: No bootstrap required, opened a new log
I20260810 18:50:38.232997 16004 raft_consensus.cc:482] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:38.232497 16006 consensus_queue.cc:237] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.233078 15962 raft_consensus.cc:348] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.233508 16004 raft_consensus.cc:504] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.233875 15962 raft_consensus.cc:374] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:38.234238 15962 raft_consensus.cc:729] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Initialized, Role: FOLLOWER
I20260810 18:50:38.234364 15959 tablet_bootstrap.cc:492] T 6df3bb273c824e68b83d4f4afdcb5b66 P 51eee958aa8948b1a46aa1641638b9ec: No bootstrap required, opened a new log
I20260810 18:50:38.234825 15959 ts_tablet_manager.cc:1397] T 6df3bb273c824e68b83d4f4afdcb5b66 P 51eee958aa8948b1a46aa1641638b9ec: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.008s
I20260810 18:50:38.235025 15962 consensus_queue.cc:260] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.235566 16004 leader_election.cc:288] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:38.237512 15962 ts_tablet_manager.cc:1428] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260810 18:50:38.237288 15959 raft_consensus.cc:348] T 6df3bb273c824e68b83d4f4afdcb5b66 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.237982 15959 raft_consensus.cc:374] T 6df3bb273c824e68b83d4f4afdcb5b66 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:38.238241 15959 raft_consensus.cc:729] T 6df3bb273c824e68b83d4f4afdcb5b66 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51eee958aa8948b1a46aa1641638b9ec, State: Initialized, Role: FOLLOWER
I20260810 18:50:38.238428 15962 tablet_bootstrap.cc:492] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670: Bootstrap starting.
I20260810 18:50:38.238744 15613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f91c7cf7b14e6ab5215e47458b4f1b" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe600335b884cae8bdd12c2ef74e670" is_pre_election: true
I20260810 18:50:38.239362 15613 raft_consensus.cc:2454] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d28250046da4d688d76be4ae6e4c826 in term 0.
I20260810 18:50:38.238896 15959 consensus_queue.cc:260] T 6df3bb273c824e68b83d4f4afdcb5b66 P 51eee958aa8948b1a46aa1641638b9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.240545 15675 leader_election.cc:302] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d28250046da4d688d76be4ae6e4c826, ffe600335b884cae8bdd12c2ef74e670; no voters:
I20260810 18:50:38.241381 15959 ts_tablet_manager.cc:1428] T 6df3bb273c824e68b83d4f4afdcb5b66 P 51eee958aa8948b1a46aa1641638b9ec: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:50:38.237375 15879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f91c7cf7b14e6ab5215e47458b4f1b" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51eee958aa8948b1a46aa1641638b9ec" is_pre_election: true
I20260810 18:50:38.242012 15879 raft_consensus.cc:2454] T 20f91c7cf7b14e6ab5215e47458b4f1b P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d28250046da4d688d76be4ae6e4c826 in term 0.
I20260810 18:50:38.241390 16004 raft_consensus.cc:2781] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:38.244325 16004 raft_consensus.cc:482] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:38.244642 15962 tablet_bootstrap.cc:654] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:38.244702 16004 raft_consensus.cc:3037] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:38.244889 15968 raft_consensus.cc:482] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:38.245332 15968 raft_consensus.cc:504] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } }
I20260810 18:50:38.247560 15968 leader_election.cc:288] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629)
I20260810 18:50:38.249115 15753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b96127e94b74666aa7a5d280228ba7d" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d28250046da4d688d76be4ae6e4c826" is_pre_election: true
I20260810 18:50:38.249718 15753 raft_consensus.cc:2454] T 8b96127e94b74666aa7a5d280228ba7d P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51eee958aa8948b1a46aa1641638b9ec in term 0.
I20260810 18:50:38.250891 15816 leader_election.cc:302] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d28250046da4d688d76be4ae6e4c826, 51eee958aa8948b1a46aa1641638b9ec; no voters:
I20260810 18:50:38.232797 15961 ts_tablet_manager.cc:1397] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.007s
I20260810 18:50:38.251744 16004 raft_consensus.cc:504] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.253106 15613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b96127e94b74666aa7a5d280228ba7d" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe600335b884cae8bdd12c2ef74e670" is_pre_election: true
I20260810 18:50:38.254328 15613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f91c7cf7b14e6ab5215e47458b4f1b" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
W20260810 18:50:38.254452 15817 leader_election.cc:341] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:38.254868 15613 raft_consensus.cc:3037] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:38.255366 15961 raft_consensus.cc:348] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.256372 15967 raft_consensus.cc:2781] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:38.256562 15961 raft_consensus.cc:374] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:38.256771 15967 raft_consensus.cc:482] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:38.256999 15961 raft_consensus.cc:729] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d28250046da4d688d76be4ae6e4c826, State: Initialized, Role: FOLLOWER
I20260810 18:50:38.257146 15967 raft_consensus.cc:3037] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:38.258142 15961 consensus_queue.cc:260] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.261502 15961 ts_tablet_manager.cc:1428] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260810 18:50:38.261574 15613 raft_consensus.cc:2454] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d28250046da4d688d76be4ae6e4c826 in term 1.
I20260810 18:50:38.262844 15675 leader_election.cc:302] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d28250046da4d688d76be4ae6e4c826, ffe600335b884cae8bdd12c2ef74e670; no voters:
I20260810 18:50:38.263754 15977 raft_consensus.cc:2781] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:38.264194 15977 raft_consensus.cc:686] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [term 1 LEADER]: Becoming Leader. State: Replica: 2d28250046da4d688d76be4ae6e4c826, State: Running, Role: LEADER
I20260810 18:50:38.264977 15977 consensus_queue.cc:237] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.266300 16004 leader_election.cc:288] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 1 election: Requested vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:38.268270 15968 raft_consensus.cc:482] T 9764b4e886a24c9db19a64075920a794 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:38.268307 15879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f91c7cf7b14e6ab5215e47458b4f1b" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
I20260810 18:50:38.268956 15879 raft_consensus.cc:3037] T 20f91c7cf7b14e6ab5215e47458b4f1b P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:38.270757 15961 tablet_bootstrap.cc:492] T 354460a175b24dcd873c370589a0f7d5 P 2d28250046da4d688d76be4ae6e4c826: Bootstrap starting.
I20260810 18:50:38.271636 15967 raft_consensus.cc:504] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } }
I20260810 18:50:38.276005 15962 tablet_bootstrap.cc:492] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670: No bootstrap required, opened a new log
I20260810 18:50:38.276484 15962 ts_tablet_manager.cc:1397] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670: Time spent bootstrapping tablet: real 0.038s user 0.014s sys 0.015s
I20260810 18:50:38.277787 15961 tablet_bootstrap.cc:654] T 354460a175b24dcd873c370589a0f7d5 P 2d28250046da4d688d76be4ae6e4c826: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:38.277848 15613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b96127e94b74666aa7a5d280228ba7d" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
W20260810 18:50:38.280634 15817 leader_election.cc:341] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:38.283001 15753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b96127e94b74666aa7a5d280228ba7d" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
I20260810 18:50:38.283792 15753 raft_consensus.cc:3037] T 8b96127e94b74666aa7a5d280228ba7d P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:38.284951 15962 raft_consensus.cc:348] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } }
I20260810 18:50:38.285622 15962 raft_consensus.cc:374] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:38.285873 15962 raft_consensus.cc:729] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Initialized, Role: FOLLOWER
I20260810 18:50:38.289207 15962 consensus_queue.cc:260] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } }
I20260810 18:50:38.291311 15753 raft_consensus.cc:2454] T 8b96127e94b74666aa7a5d280228ba7d P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51eee958aa8948b1a46aa1641638b9ec in term 1.
I20260810 18:50:38.292510 15816 leader_election.cc:302] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d28250046da4d688d76be4ae6e4c826, 51eee958aa8948b1a46aa1641638b9ec; no voters: ffe600335b884cae8bdd12c2ef74e670
I20260810 18:50:38.294720 15962 ts_tablet_manager.cc:1428] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260810 18:50:38.268949 15968 raft_consensus.cc:504] T 9764b4e886a24c9db19a64075920a794 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.286276 15417 catalog_manager.cc:5495] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 reported cstate change: term changed from 0 to 1, leader changed from <none> to ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193). New cstate: current_term: 1 leader_uuid: "ffe600335b884cae8bdd12c2ef74e670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:38.296777 15968 leader_election.cc:288] T 9764b4e886a24c9db19a64075920a794 P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629)
I20260810 18:50:38.297406 15968 raft_consensus.cc:482] T 6df3bb273c824e68b83d4f4afdcb5b66 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:38.298156 15967 leader_election.cc:288] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 election: Requested vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629)
I20260810 18:50:38.297919 15968 raft_consensus.cc:504] T 6df3bb273c824e68b83d4f4afdcb5b66 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.298663 15967 raft_consensus.cc:2781] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:38.299120 15967 raft_consensus.cc:686] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [term 1 LEADER]: Becoming Leader. State: Replica: 51eee958aa8948b1a46aa1641638b9ec, State: Running, Role: LEADER
I20260810 18:50:38.300140 15613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9764b4e886a24c9db19a64075920a794" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe600335b884cae8bdd12c2ef74e670" is_pre_election: true
I20260810 18:50:38.301409 15879 raft_consensus.cc:2454] T 20f91c7cf7b14e6ab5215e47458b4f1b P 51eee958aa8948b1a46aa1641638b9ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d28250046da4d688d76be4ae6e4c826 in term 1.
I20260810 18:50:38.301857 15612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6df3bb273c824e68b83d4f4afdcb5b66" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe600335b884cae8bdd12c2ef74e670" is_pre_election: true
I20260810 18:50:38.302537 15612 raft_consensus.cc:2454] T 6df3bb273c824e68b83d4f4afdcb5b66 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51eee958aa8948b1a46aa1641638b9ec in term 0.
I20260810 18:50:38.304170 15817 leader_election.cc:302] T 6df3bb273c824e68b83d4f4afdcb5b66 P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51eee958aa8948b1a46aa1641638b9ec, ffe600335b884cae8bdd12c2ef74e670; no voters:
I20260810 18:50:38.305256 15968 leader_election.cc:288] T 6df3bb273c824e68b83d4f4afdcb5b66 P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629)
I20260810 18:50:38.306003 16043 raft_consensus.cc:482] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:38.306541 16043 raft_consensus.cc:504] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.314182 15968 raft_consensus.cc:2781] T 6df3bb273c824e68b83d4f4afdcb5b66 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:38.314615 15968 raft_consensus.cc:482] T 6df3bb273c824e68b83d4f4afdcb5b66 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:38.315668 15967 consensus_queue.cc:237] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } }
I20260810 18:50:38.316382 15962 tablet_bootstrap.cc:492] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670: Bootstrap starting.
I20260810 18:50:38.319698 16048 raft_consensus.cc:482] T 2813a19d92094a648be20c4c20f713fa P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:38.320057 15612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354460a175b24dcd873c370589a0f7d5" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe600335b884cae8bdd12c2ef74e670" is_pre_election: true
I20260810 18:50:38.320183 15752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6df3bb273c824e68b83d4f4afdcb5b66" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d28250046da4d688d76be4ae6e4c826" is_pre_election: true
I20260810 18:50:38.320323 16048 raft_consensus.cc:504] T 2813a19d92094a648be20c4c20f713fa P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.325829 15961 tablet_bootstrap.cc:492] T 354460a175b24dcd873c370589a0f7d5 P 2d28250046da4d688d76be4ae6e4c826: No bootstrap required, opened a new log
I20260810 18:50:38.326494 15961 ts_tablet_manager.cc:1397] T 354460a175b24dcd873c370589a0f7d5 P 2d28250046da4d688d76be4ae6e4c826: Time spent bootstrapping tablet: real 0.056s user 0.035s sys 0.017s
I20260810 18:50:38.321259 15752 raft_consensus.cc:2454] T 6df3bb273c824e68b83d4f4afdcb5b66 P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51eee958aa8948b1a46aa1641638b9ec in term 0.
W20260810 18:50:38.322114 15817 leader_election.cc:341] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:38.314951 15968 raft_consensus.cc:3037] T 6df3bb273c824e68b83d4f4afdcb5b66 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:38.323017 16004 raft_consensus.cc:482] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:38.327749 16004 raft_consensus.cc:504] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.329815 16004 leader_election.cc:288] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:38.330549 15961 raft_consensus.cc:348] T 354460a175b24dcd873c370589a0f7d5 P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.331341 15961 raft_consensus.cc:374] T 354460a175b24dcd873c370589a0f7d5 P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:38.331660 15961 raft_consensus.cc:729] T 354460a175b24dcd873c370589a0f7d5 P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d28250046da4d688d76be4ae6e4c826, State: Initialized, Role: FOLLOWER
I20260810 18:50:38.332335 15961 consensus_queue.cc:260] T 354460a175b24dcd873c370589a0f7d5 P 2d28250046da4d688d76be4ae6e4c826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.333159 15612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e61e9bbff154604bce6712d94c73e90" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe600335b884cae8bdd12c2ef74e670" is_pre_election: true
I20260810 18:50:38.320972 15753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9764b4e886a24c9db19a64075920a794" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d28250046da4d688d76be4ae6e4c826" is_pre_election: true
I20260810 18:50:38.334694 15961 ts_tablet_manager.cc:1428] T 354460a175b24dcd873c370589a0f7d5 P 2d28250046da4d688d76be4ae6e4c826: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260810 18:50:38.335551 15753 raft_consensus.cc:2379] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 51eee958aa8948b1a46aa1641638b9ec in current term 1: Already voted for candidate ffe600335b884cae8bdd12c2ef74e670 in this term.
I20260810 18:50:38.338145 16048 leader_election.cc:288] T 2813a19d92094a648be20c4c20f713fa P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629)
I20260810 18:50:38.336372 15879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e61e9bbff154604bce6712d94c73e90" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51eee958aa8948b1a46aa1641638b9ec" is_pre_election: true
I20260810 18:50:38.339048 16050 raft_consensus.cc:482] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:38.339742 15879 raft_consensus.cc:2379] T 8e61e9bbff154604bce6712d94c73e90 P 51eee958aa8948b1a46aa1641638b9ec [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2d28250046da4d688d76be4ae6e4c826 in current term 1: Already voted for candidate ffe600335b884cae8bdd12c2ef74e670 in this term.
I20260810 18:50:38.335704 15961 tablet_bootstrap.cc:492] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826: Bootstrap starting.
I20260810 18:50:38.338498 16006 raft_consensus.cc:482] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:38.339794 15612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2813a19d92094a648be20c4c20f713fa" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe600335b884cae8bdd12c2ef74e670" is_pre_election: true
I20260810 18:50:38.340822 15612 raft_consensus.cc:2379] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 51eee958aa8948b1a46aa1641638b9ec in current term 1: Already voted for candidate 2d28250046da4d688d76be4ae6e4c826 in this term.
I20260810 18:50:38.340435 16006 raft_consensus.cc:504] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.339556 16050 raft_consensus.cc:504] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.338104 15816 leader_election.cc:302] T 9764b4e886a24c9db19a64075920a794 P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51eee958aa8948b1a46aa1641638b9ec; no voters: 2d28250046da4d688d76be4ae6e4c826, ffe600335b884cae8bdd12c2ef74e670
I20260810 18:50:38.343998 15967 raft_consensus.cc:3037] T 9764b4e886a24c9db19a64075920a794 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:38.344259 15673 leader_election.cc:302] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d28250046da4d688d76be4ae6e4c826; no voters: 51eee958aa8948b1a46aa1641638b9ec, ffe600335b884cae8bdd12c2ef74e670
I20260810 18:50:38.344738 15752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e5673440be4239a048129ffad7a53c" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d28250046da4d688d76be4ae6e4c826" is_pre_election: true
W20260810 18:50:38.346004 15816 leader_election.cc:341] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:38.346400 16043 leader_election.cc:288] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629)
I20260810 18:50:38.347432 15752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e5673440be4239a048129ffad7a53c" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d28250046da4d688d76be4ae6e4c826" is_pre_election: true
I20260810 18:50:38.347869 15750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354460a175b24dcd873c370589a0f7d5" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d28250046da4d688d76be4ae6e4c826" is_pre_election: true
I20260810 18:50:38.348541 15750 raft_consensus.cc:2454] T 354460a175b24dcd873c370589a0f7d5 P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51eee958aa8948b1a46aa1641638b9ec in term 0.
I20260810 18:50:38.345805 16006 leader_election.cc:288] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:38.350910 15816 leader_election.cc:302] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d28250046da4d688d76be4ae6e4c826, 51eee958aa8948b1a46aa1641638b9ec; no voters: ffe600335b884cae8bdd12c2ef74e670
I20260810 18:50:38.351608 15962 tablet_bootstrap.cc:654] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:38.352133 15967 raft_consensus.cc:2726] T 9764b4e886a24c9db19a64075920a794 P 51eee958aa8948b1a46aa1641638b9ec [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:50:38.352727 15967 raft_consensus.cc:2781] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:38.353091 15967 raft_consensus.cc:482] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:38.353410 15967 raft_consensus.cc:3037] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:38.353529 15417 catalog_manager.cc:5495] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195). New cstate: current_term: 1 leader_uuid: "51eee958aa8948b1a46aa1641638b9ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:38.355921 15612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e5673440be4239a048129ffad7a53c" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe600335b884cae8bdd12c2ef74e670" is_pre_election: true
W20260810 18:50:38.351771 15537 leader_election.cc:341] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:38.350816 15879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e5673440be4239a048129ffad7a53c" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51eee958aa8948b1a46aa1641638b9ec" is_pre_election: true
I20260810 18:50:38.351549 16050 leader_election.cc:288] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629)
I20260810 18:50:38.357625 15879 raft_consensus.cc:2454] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffe600335b884cae8bdd12c2ef74e670 in term 0.
I20260810 18:50:38.359978 15967 raft_consensus.cc:504] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.361230 15961 tablet_bootstrap.cc:654] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:38.363157 15613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354460a175b24dcd873c370589a0f7d5" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
W20260810 18:50:38.364339 15817 leader_election.cc:341] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:38.365497 15967 leader_election.cc:288] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 election: Requested vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629)
I20260810 18:50:38.351421 15977 raft_consensus.cc:3037] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:38.356742 15612 raft_consensus.cc:2454] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51eee958aa8948b1a46aa1641638b9ec in term 0.
I20260810 18:50:38.370615 15418 catalog_manager.cc:5495] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194). New cstate: current_term: 1 leader_uuid: "2d28250046da4d688d76be4ae6e4c826" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:38.368017 15750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354460a175b24dcd873c370589a0f7d5" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
I20260810 18:50:38.375914 15817 leader_election.cc:302] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51eee958aa8948b1a46aa1641638b9ec, ffe600335b884cae8bdd12c2ef74e670; no voters: 2d28250046da4d688d76be4ae6e4c826
I20260810 18:50:38.378160 15750 raft_consensus.cc:3037] T 354460a175b24dcd873c370589a0f7d5 P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:38.374513 15968 raft_consensus.cc:504] T 6df3bb273c824e68b83d4f4afdcb5b66 P 51eee958aa8948b1a46aa1641638b9ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.357893 15753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2813a19d92094a648be20c4c20f713fa" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d28250046da4d688d76be4ae6e4c826" is_pre_election: true
I20260810 18:50:38.366926 15537 leader_election.cc:302] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51eee958aa8948b1a46aa1641638b9ec, ffe600335b884cae8bdd12c2ef74e670; no voters: 2d28250046da4d688d76be4ae6e4c826
I20260810 18:50:38.382490 15967 raft_consensus.cc:2781] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:38.382872 15967 raft_consensus.cc:482] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:38.383009 16006 raft_consensus.cc:2781] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:38.383282 15967 raft_consensus.cc:3037] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:38.383464 16006 raft_consensus.cc:482] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:38.383862 16006 raft_consensus.cc:3037] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:38.383992 15977 raft_consensus.cc:2726] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:50:38.384256 15613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6df3bb273c824e68b83d4f4afdcb5b66" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
I20260810 18:50:38.384918 15613 raft_consensus.cc:3037] T 6df3bb273c824e68b83d4f4afdcb5b66 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:38.385823 15968 leader_election.cc:288] T 6df3bb273c824e68b83d4f4afdcb5b66 P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 election: Requested vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629)
I20260810 18:50:38.391139 15750 raft_consensus.cc:2454] T 354460a175b24dcd873c370589a0f7d5 P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51eee958aa8948b1a46aa1641638b9ec in term 1.
I20260810 18:50:38.396430 15816 leader_election.cc:302] T 2813a19d92094a648be20c4c20f713fa P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51eee958aa8948b1a46aa1641638b9ec; no voters: 2d28250046da4d688d76be4ae6e4c826, ffe600335b884cae8bdd12c2ef74e670
I20260810 18:50:38.397209 15968 raft_consensus.cc:3037] T 2813a19d92094a648be20c4c20f713fa P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:38.398695 15613 raft_consensus.cc:2454] T 6df3bb273c824e68b83d4f4afdcb5b66 P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51eee958aa8948b1a46aa1641638b9ec in term 1.
I20260810 18:50:38.392809 15967 raft_consensus.cc:504] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.395643 15753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6df3bb273c824e68b83d4f4afdcb5b66" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
I20260810 18:50:38.401582 15816 leader_election.cc:302] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d28250046da4d688d76be4ae6e4c826, 51eee958aa8948b1a46aa1641638b9ec; no voters: ffe600335b884cae8bdd12c2ef74e670
I20260810 18:50:38.402171 15962 tablet_bootstrap.cc:492] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670: No bootstrap required, opened a new log
I20260810 18:50:38.402420 16043 raft_consensus.cc:2781] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:38.400220 15817 leader_election.cc:302] T 6df3bb273c824e68b83d4f4afdcb5b66 P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51eee958aa8948b1a46aa1641638b9ec, ffe600335b884cae8bdd12c2ef74e670; no voters:
I20260810 18:50:38.403398 16048 raft_consensus.cc:2781] T 6df3bb273c824e68b83d4f4afdcb5b66 P 51eee958aa8948b1a46aa1641638b9ec [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:38.403821 16048 raft_consensus.cc:686] T 6df3bb273c824e68b83d4f4afdcb5b66 P 51eee958aa8948b1a46aa1641638b9ec [term 1 LEADER]: Becoming Leader. State: Replica: 51eee958aa8948b1a46aa1641638b9ec, State: Running, Role: LEADER
I20260810 18:50:38.404021 15977 consensus_queue.cc:1035] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:50:38.404659 16048 consensus_queue.cc:237] T 6df3bb273c824e68b83d4f4afdcb5b66 P 51eee958aa8948b1a46aa1641638b9ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.405331 15613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e5673440be4239a048129ffad7a53c" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
I20260810 18:50:38.402614 15962 ts_tablet_manager.cc:1397] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670: Time spent bootstrapping tablet: real 0.086s user 0.017s sys 0.013s
I20260810 18:50:38.403834 15753 raft_consensus.cc:3037] T 6df3bb273c824e68b83d4f4afdcb5b66 P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:38.396021 16006 raft_consensus.cc:504] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.402858 16043 raft_consensus.cc:686] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 1 LEADER]: Becoming Leader. State: Replica: 51eee958aa8948b1a46aa1641638b9ec, State: Running, Role: LEADER
I20260810 18:50:38.414216 15967 leader_election.cc:288] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 election: Requested vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629)
I20260810 18:50:38.414645 16043 consensus_queue.cc:237] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.417073 15968 raft_consensus.cc:2726] T 2813a19d92094a648be20c4c20f713fa P 51eee958aa8948b1a46aa1641638b9ec [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:50:38.417954 16050 raft_consensus.cc:482] T cd4103b9a0414848b8ad7e116bf4ded8 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:38.420804 15613 raft_consensus.cc:2379] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 51eee958aa8948b1a46aa1641638b9ec in current term 1: Already voted for candidate ffe600335b884cae8bdd12c2ef74e670 in this term.
I20260810 18:50:38.421442 15961 tablet_bootstrap.cc:492] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826: No bootstrap required, opened a new log
I20260810 18:50:38.422047 15961 ts_tablet_manager.cc:1397] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826: Time spent bootstrapping tablet: real 0.086s user 0.023s sys 0.016s
I20260810 18:50:38.420252 16050 raft_consensus.cc:504] T cd4103b9a0414848b8ad7e116bf4ded8 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.428758 15961 raft_consensus.cc:348] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.429647 15961 raft_consensus.cc:374] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:38.429983 15961 raft_consensus.cc:729] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d28250046da4d688d76be4ae6e4c826, State: Initialized, Role: FOLLOWER
I20260810 18:50:38.431504 15750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e5673440be4239a048129ffad7a53c" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
I20260810 18:50:38.431818 15613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd4103b9a0414848b8ad7e116bf4ded8" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe600335b884cae8bdd12c2ef74e670" is_pre_election: true
W20260810 18:50:38.433274 15817 leader_election.cc:341] T cd4103b9a0414848b8ad7e116bf4ded8 P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:38.435451 16050 leader_election.cc:288] T cd4103b9a0414848b8ad7e116bf4ded8 P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629)
I20260810 18:50:38.436023 15962 raft_consensus.cc:348] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.436805 15962 raft_consensus.cc:374] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:38.436388 15961 consensus_queue.cc:260] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.437103 15962 raft_consensus.cc:729] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Initialized, Role: FOLLOWER
I20260810 18:50:38.437294 15750 raft_consensus.cc:3037] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:38.437704 15752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd4103b9a0414848b8ad7e116bf4ded8" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d28250046da4d688d76be4ae6e4c826" is_pre_election: true
I20260810 18:50:38.437739 15962 consensus_queue.cc:260] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
W20260810 18:50:38.439257 15816 leader_election.cc:341] T cd4103b9a0414848b8ad7e116bf4ded8 P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:38.439671 15816 leader_election.cc:302] T cd4103b9a0414848b8ad7e116bf4ded8 P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51eee958aa8948b1a46aa1641638b9ec; no voters: 2d28250046da4d688d76be4ae6e4c826, ffe600335b884cae8bdd12c2ef74e670
I20260810 18:50:38.440104 15962 ts_tablet_manager.cc:1428] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670: Time spent starting tablet: real 0.028s user 0.004s sys 0.002s
I20260810 18:50:38.440987 15962 tablet_bootstrap.cc:492] T cd4103b9a0414848b8ad7e116bf4ded8 P ffe600335b884cae8bdd12c2ef74e670: Bootstrap starting.
I20260810 18:50:38.441025 15753 raft_consensus.cc:2454] T 6df3bb273c824e68b83d4f4afdcb5b66 P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51eee958aa8948b1a46aa1641638b9ec in term 1.
I20260810 18:50:38.444416 15750 raft_consensus.cc:2454] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51eee958aa8948b1a46aa1641638b9ec in term 1.
I20260810 18:50:38.444950 16004 raft_consensus.cc:482] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:38.445991 16006 leader_election.cc:288] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 1 election: Requested vote from peers 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:38.445317 15418 catalog_manager.cc:5495] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195). New cstate: current_term: 1 leader_uuid: "51eee958aa8948b1a46aa1641638b9ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } health_report { overall_health: HEALTHY } } }
I20260810 18:50:38.446287 15977 consensus_queue.cc:1035] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:50:38.445356 16004 raft_consensus.cc:504] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.476934 15612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e5673440be4239a048129ffad7a53c" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe600335b884cae8bdd12c2ef74e670" is_pre_election: true
I20260810 18:50:38.463666 16048 raft_consensus.cc:2726] T cd4103b9a0414848b8ad7e116bf4ded8 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:50:38.477661 15612 raft_consensus.cc:2454] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d28250046da4d688d76be4ae6e4c826 in term 1.
I20260810 18:50:38.478768 15675 leader_election.cc:302] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d28250046da4d688d76be4ae6e4c826, ffe600335b884cae8bdd12c2ef74e670; no voters:
I20260810 18:50:38.479568 15962 tablet_bootstrap.cc:654] T cd4103b9a0414848b8ad7e116bf4ded8 P ffe600335b884cae8bdd12c2ef74e670: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:38.481141 15879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e5673440be4239a048129ffad7a53c" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
I20260810 18:50:38.482051 15879 raft_consensus.cc:2379] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ffe600335b884cae8bdd12c2ef74e670 in current term 1: Already voted for candidate 51eee958aa8948b1a46aa1641638b9ec in this term.
I20260810 18:50:38.482838 15753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e5673440be4239a048129ffad7a53c" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
I20260810 18:50:38.484045 16004 leader_election.cc:288] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:38.484676 16004 raft_consensus.cc:2781] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:50:38.485163 16004 raft_consensus.cc:482] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:38.485237 15879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e5673440be4239a048129ffad7a53c" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51eee958aa8948b1a46aa1641638b9ec" is_pre_election: true
I20260810 18:50:38.485977 15879 raft_consensus.cc:2454] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d28250046da4d688d76be4ae6e4c826 in term 1.
I20260810 18:50:38.485824 16004 raft_consensus.cc:3037] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:38.493057 15537 leader_election.cc:302] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ffe600335b884cae8bdd12c2ef74e670; no voters: 2d28250046da4d688d76be4ae6e4c826, 51eee958aa8948b1a46aa1641638b9ec
I20260810 18:50:38.493934 16006 raft_consensus.cc:2726] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 18:50:38.495731 15961 ts_tablet_manager.cc:1428] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826: Time spent starting tablet: real 0.073s user 0.011s sys 0.001s
I20260810 18:50:38.496901 15961 tablet_bootstrap.cc:492] T cd4103b9a0414848b8ad7e116bf4ded8 P 2d28250046da4d688d76be4ae6e4c826: Bootstrap starting.
I20260810 18:50:38.498751 15816 leader_election.cc:302] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d28250046da4d688d76be4ae6e4c826, 51eee958aa8948b1a46aa1641638b9ec; no voters: ffe600335b884cae8bdd12c2ef74e670
I20260810 18:50:38.500046 16048 raft_consensus.cc:2781] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:38.500737 16048 raft_consensus.cc:686] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 1 LEADER]: Becoming Leader. State: Replica: 51eee958aa8948b1a46aa1641638b9ec, State: Running, Role: LEADER
I20260810 18:50:38.501624 16048 consensus_queue.cc:237] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.514787 16004 raft_consensus.cc:504] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.515789 15961 tablet_bootstrap.cc:654] T cd4103b9a0414848b8ad7e116bf4ded8 P 2d28250046da4d688d76be4ae6e4c826: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:38.517112 16004 leader_election.cc:288] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 2 election: Requested vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:38.519348 15879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e5673440be4239a048129ffad7a53c" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
I20260810 18:50:38.517270 15962 tablet_bootstrap.cc:492] T cd4103b9a0414848b8ad7e116bf4ded8 P ffe600335b884cae8bdd12c2ef74e670: No bootstrap required, opened a new log
I20260810 18:50:38.523456 15962 ts_tablet_manager.cc:1397] T cd4103b9a0414848b8ad7e116bf4ded8 P ffe600335b884cae8bdd12c2ef74e670: Time spent bootstrapping tablet: real 0.083s user 0.006s sys 0.006s
I20260810 18:50:38.531324 16006 consensus_queue.cc:1035] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:50:38.526535 15962 raft_consensus.cc:348] T cd4103b9a0414848b8ad7e116bf4ded8 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.535964 15962 raft_consensus.cc:374] T cd4103b9a0414848b8ad7e116bf4ded8 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:38.535923 15613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e5673440be4239a048129ffad7a53c" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
I20260810 18:50:38.536309 15962 raft_consensus.cc:729] T cd4103b9a0414848b8ad7e116bf4ded8 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Initialized, Role: FOLLOWER
I20260810 18:50:38.536664 15613 raft_consensus.cc:3037] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:38.536918 15962 consensus_queue.cc:260] T cd4103b9a0414848b8ad7e116bf4ded8 P ffe600335b884cae8bdd12c2ef74e670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.537534 15961 tablet_bootstrap.cc:492] T cd4103b9a0414848b8ad7e116bf4ded8 P 2d28250046da4d688d76be4ae6e4c826: No bootstrap required, opened a new log
I20260810 18:50:38.538092 15961 ts_tablet_manager.cc:1397] T cd4103b9a0414848b8ad7e116bf4ded8 P 2d28250046da4d688d76be4ae6e4c826: Time spent bootstrapping tablet: real 0.041s user 0.007s sys 0.009s
I20260810 18:50:38.541275 15961 raft_consensus.cc:348] T cd4103b9a0414848b8ad7e116bf4ded8 P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.542132 15961 raft_consensus.cc:374] T cd4103b9a0414848b8ad7e116bf4ded8 P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:38.542467 15961 raft_consensus.cc:729] T cd4103b9a0414848b8ad7e116bf4ded8 P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d28250046da4d688d76be4ae6e4c826, State: Initialized, Role: FOLLOWER
I20260810 18:50:38.544759 15962 ts_tablet_manager.cc:1428] T cd4103b9a0414848b8ad7e116bf4ded8 P ffe600335b884cae8bdd12c2ef74e670: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20260810 18:50:38.550359 15961 consensus_queue.cc:260] T cd4103b9a0414848b8ad7e116bf4ded8 P 2d28250046da4d688d76be4ae6e4c826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.554231 15961 ts_tablet_manager.cc:1428] T cd4103b9a0414848b8ad7e116bf4ded8 P 2d28250046da4d688d76be4ae6e4c826: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20260810 18:50:38.556515 15613 raft_consensus.cc:2454] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d28250046da4d688d76be4ae6e4c826 in term 2.
I20260810 18:50:38.557636 15675 leader_election.cc:302] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d28250046da4d688d76be4ae6e4c826, ffe600335b884cae8bdd12c2ef74e670; no voters: 51eee958aa8948b1a46aa1641638b9ec
I20260810 18:50:38.558915 16003 raft_consensus.cc:2781] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:50:38.559412 16003 raft_consensus.cc:686] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [term 2 LEADER]: Becoming Leader. State: Replica: 2d28250046da4d688d76be4ae6e4c826, State: Running, Role: LEADER
I20260810 18:50:38.560204 16003 consensus_queue.cc:237] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.562850 16006 raft_consensus.cc:482] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:38.563374 16006 raft_consensus.cc:504] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.565688 16006 leader_election.cc:288] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:38.567490 15750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354460a175b24dcd873c370589a0f7d5" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d28250046da4d688d76be4ae6e4c826" is_pre_election: true
I20260810 18:50:38.568017 15878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354460a175b24dcd873c370589a0f7d5" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51eee958aa8948b1a46aa1641638b9ec" is_pre_election: true
I20260810 18:50:38.568315 15750 raft_consensus.cc:2379] T 354460a175b24dcd873c370589a0f7d5 P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ffe600335b884cae8bdd12c2ef74e670 in current term 1: Already voted for candidate 51eee958aa8948b1a46aa1641638b9ec in this term.
I20260810 18:50:38.571607 15537 leader_election.cc:302] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ffe600335b884cae8bdd12c2ef74e670; no voters: 2d28250046da4d688d76be4ae6e4c826, 51eee958aa8948b1a46aa1641638b9ec
I20260810 18:50:38.573167 15979 raft_consensus.cc:3037] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:38.576936 15417 catalog_manager.cc:5495] T 6df3bb273c824e68b83d4f4afdcb5b66 P 51eee958aa8948b1a46aa1641638b9ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195). New cstate: current_term: 1 leader_uuid: "51eee958aa8948b1a46aa1641638b9ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } health_report { overall_health: HEALTHY } } }
I20260810 18:50:38.578886 15417 catalog_manager.cc:5495] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195). New cstate: current_term: 1 leader_uuid: "51eee958aa8948b1a46aa1641638b9ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } health_report { overall_health: HEALTHY } } }
I20260810 18:50:38.579962 15979 raft_consensus.cc:2726] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:50:38.583832 16077 consensus_queue.cc:1035] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:50:38.590371 15418 catalog_manager.cc:5495] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 reported cstate change: term changed from 1 to 2, leader changed from 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195) to 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194). New cstate: current_term: 2 leader_uuid: "2d28250046da4d688d76be4ae6e4c826" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:38.594377 16003 consensus_queue.cc:1035] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:50:38.616762 16003 consensus_queue.cc:1035] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:50:38.624969 16077 consensus_queue.cc:1035] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:50:38.643316 15979 consensus_queue.cc:1035] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:50:38.673928 16060 raft_consensus.cc:482] T cd4103b9a0414848b8ad7e116bf4ded8 P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:38.674417 16060 raft_consensus.cc:504] T cd4103b9a0414848b8ad7e116bf4ded8 P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.676265 16060 leader_election.cc:288] T cd4103b9a0414848b8ad7e116bf4ded8 P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:38.677125 15613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd4103b9a0414848b8ad7e116bf4ded8" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe600335b884cae8bdd12c2ef74e670" is_pre_election: true
I20260810 18:50:38.677380 15878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd4103b9a0414848b8ad7e116bf4ded8" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51eee958aa8948b1a46aa1641638b9ec" is_pre_election: true
I20260810 18:50:38.677714 15613 raft_consensus.cc:2454] T cd4103b9a0414848b8ad7e116bf4ded8 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d28250046da4d688d76be4ae6e4c826 in term 0.
I20260810 18:50:38.678048 15878 raft_consensus.cc:2454] T cd4103b9a0414848b8ad7e116bf4ded8 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d28250046da4d688d76be4ae6e4c826 in term 0.
I20260810 18:50:38.678781 15675 leader_election.cc:302] T cd4103b9a0414848b8ad7e116bf4ded8 P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d28250046da4d688d76be4ae6e4c826, ffe600335b884cae8bdd12c2ef74e670; no voters:
I20260810 18:50:38.680022 16060 raft_consensus.cc:2781] T cd4103b9a0414848b8ad7e116bf4ded8 P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:38.680363 16060 raft_consensus.cc:482] T cd4103b9a0414848b8ad7e116bf4ded8 P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:38.680693 16060 raft_consensus.cc:3037] T cd4103b9a0414848b8ad7e116bf4ded8 P 2d28250046da4d688d76be4ae6e4c826 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:38.682114 16003 consensus_queue.cc:1035] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:50:38.687290 16060 raft_consensus.cc:504] T cd4103b9a0414848b8ad7e116bf4ded8 P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.689347 16060 leader_election.cc:288] T cd4103b9a0414848b8ad7e116bf4ded8 P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 1 election: Requested vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:38.690353 15878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd4103b9a0414848b8ad7e116bf4ded8" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
I20260810 18:50:38.690927 15878 raft_consensus.cc:3037] T cd4103b9a0414848b8ad7e116bf4ded8 P 51eee958aa8948b1a46aa1641638b9ec [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:38.693389 15977 consensus_queue.cc:1035] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:50:38.698448 15613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd4103b9a0414848b8ad7e116bf4ded8" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
I20260810 18:50:38.699867 15613 raft_consensus.cc:3037] T cd4103b9a0414848b8ad7e116bf4ded8 P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:38.700845 15878 raft_consensus.cc:2454] T cd4103b9a0414848b8ad7e116bf4ded8 P 51eee958aa8948b1a46aa1641638b9ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d28250046da4d688d76be4ae6e4c826 in term 1.
I20260810 18:50:38.701965 15673 leader_election.cc:302] T cd4103b9a0414848b8ad7e116bf4ded8 P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d28250046da4d688d76be4ae6e4c826, 51eee958aa8948b1a46aa1641638b9ec; no voters:
I20260810 18:50:38.706622 15977 raft_consensus.cc:2781] T cd4103b9a0414848b8ad7e116bf4ded8 P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:38.707255 15977 raft_consensus.cc:686] T cd4103b9a0414848b8ad7e116bf4ded8 P 2d28250046da4d688d76be4ae6e4c826 [term 1 LEADER]: Becoming Leader. State: Replica: 2d28250046da4d688d76be4ae6e4c826, State: Running, Role: LEADER
I20260810 18:50:38.708060 15977 consensus_queue.cc:237] T cd4103b9a0414848b8ad7e116bf4ded8 P 2d28250046da4d688d76be4ae6e4c826 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.709167 15613 raft_consensus.cc:2454] T cd4103b9a0414848b8ad7e116bf4ded8 P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d28250046da4d688d76be4ae6e4c826 in term 1.
I20260810 18:50:38.716090 15417 catalog_manager.cc:5495] T cd4103b9a0414848b8ad7e116bf4ded8 P 2d28250046da4d688d76be4ae6e4c826 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194). New cstate: current_term: 1 leader_uuid: "2d28250046da4d688d76be4ae6e4c826" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:38.748051 15878 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8b96127e94b74666aa7a5d280228ba7d"
dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
mode: GRACEFUL
new_leader_uuid: "ffe600335b884cae8bdd12c2ef74e670"
from {username='slave'} at 127.0.0.1:35188
I20260810 18:50:38.748769 15878 raft_consensus.cc:595] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [term 1 LEADER]: Received request to transfer leadership to TS ffe600335b884cae8bdd12c2ef74e670
I20260810 18:50:38.764266 15979 raft_consensus.cc:482] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:38.764446 15750 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4f6b952757814de89b38a47f3942ee7e"
dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
mode: GRACEFUL
new_leader_uuid: "ffe600335b884cae8bdd12c2ef74e670"
from {username='slave'} at 127.0.0.1:59512
I20260810 18:50:38.765106 15750 raft_consensus.cc:595] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [term 1 LEADER]: Received request to transfer leadership to TS ffe600335b884cae8bdd12c2ef74e670
I20260810 18:50:38.764812 15979 raft_consensus.cc:504] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } }
I20260810 18:50:38.766782 15979 leader_election.cc:288] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891), 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629)
I20260810 18:50:38.767500 15878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b96127e94b74666aa7a5d280228ba7d" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51eee958aa8948b1a46aa1641638b9ec" is_pre_election: true
I20260810 18:50:38.767845 15750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b96127e94b74666aa7a5d280228ba7d" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d28250046da4d688d76be4ae6e4c826" is_pre_election: true
I20260810 18:50:38.768690 15750 raft_consensus.cc:2379] T 8b96127e94b74666aa7a5d280228ba7d P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ffe600335b884cae8bdd12c2ef74e670 in current term 1: Already voted for candidate 51eee958aa8948b1a46aa1641638b9ec in this term.
I20260810 18:50:38.769574 15537 leader_election.cc:302] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ffe600335b884cae8bdd12c2ef74e670; no voters: 2d28250046da4d688d76be4ae6e4c826, 51eee958aa8948b1a46aa1641638b9ec
I20260810 18:50:38.770228 15979 raft_consensus.cc:3037] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:38.770625 15750 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "20f91c7cf7b14e6ab5215e47458b4f1b"
dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
mode: GRACEFUL
new_leader_uuid: "ffe600335b884cae8bdd12c2ef74e670"
from {username='slave'} at 127.0.0.1:59512
I20260810 18:50:38.771160 15750 raft_consensus.cc:595] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [term 1 LEADER]: Received request to transfer leadership to TS ffe600335b884cae8bdd12c2ef74e670
I20260810 18:50:38.776268 15979 raft_consensus.cc:2726] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:50:38.776394 15878 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "354460a175b24dcd873c370589a0f7d5"
dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
mode: GRACEFUL
new_leader_uuid: "ffe600335b884cae8bdd12c2ef74e670"
from {username='slave'} at 127.0.0.1:35188
I20260810 18:50:38.777045 15878 raft_consensus.cc:595] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 1 LEADER]: Received request to transfer leadership to TS ffe600335b884cae8bdd12c2ef74e670
I20260810 18:50:38.781708 15750 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e7e5673440be4239a048129ffad7a53c"
dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
mode: GRACEFUL
new_leader_uuid: "ffe600335b884cae8bdd12c2ef74e670"
from {username='slave'} at 127.0.0.1:59512
I20260810 18:50:38.782276 15750 raft_consensus.cc:595] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [term 2 LEADER]: Received request to transfer leadership to TS ffe600335b884cae8bdd12c2ef74e670
I20260810 18:50:38.791345 16048 consensus_queue.cc:1035] T 6df3bb273c824e68b83d4f4afdcb5b66 P 51eee958aa8948b1a46aa1641638b9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:50:38.804947 15613 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8e61e9bbff154604bce6712d94c73e90"
dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
mode: GRACEFUL
new_leader_uuid: "2d28250046da4d688d76be4ae6e4c826"
from {username='slave'} at 127.0.0.1:60336
I20260810 18:50:38.805563 15613 raft_consensus.cc:595] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [term 1 LEADER]: Received request to transfer leadership to TS 2d28250046da4d688d76be4ae6e4c826
I20260810 18:50:38.806576 16048 consensus_queue.cc:1035] T 6df3bb273c824e68b83d4f4afdcb5b66 P 51eee958aa8948b1a46aa1641638b9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:50:38.809175 16043 consensus_queue.cc:1035] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:50:38.814471 15612 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9764b4e886a24c9db19a64075920a794"
dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
mode: GRACEFUL
new_leader_uuid: "2d28250046da4d688d76be4ae6e4c826"
from {username='slave'} at 127.0.0.1:60336
I20260810 18:50:38.815122 15612 raft_consensus.cc:595] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [term 1 LEADER]: Received request to transfer leadership to TS 2d28250046da4d688d76be4ae6e4c826
I20260810 18:50:38.820307 16050 raft_consensus.cc:979] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec: : Instructing follower ffe600335b884cae8bdd12c2ef74e670 to start an election
I20260810 18:50:38.820853 16050 raft_consensus.cc:1067] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [term 1 LEADER]: Signalling peer ffe600335b884cae8bdd12c2ef74e670 to start an election
I20260810 18:50:38.824287 15612 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8b96127e94b74666aa7a5d280228ba7d"
dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
from {username='slave'} at 127.0.0.1:60318
I20260810 18:50:38.825081 15612 raft_consensus.cc:482] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:38.825553 15612 raft_consensus.cc:3037] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:38.827795 16050 consensus_queue.cc:1035] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:50:38.835345 15612 raft_consensus.cc:504] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } }
I20260810 18:50:38.837350 15612 leader_election.cc:288] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 2 election: Requested vote from peers 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891), 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629)
I20260810 18:50:38.838508 15878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b96127e94b74666aa7a5d280228ba7d" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
I20260810 18:50:38.839193 15878 raft_consensus.cc:3032] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:50:38.839534 15878 raft_consensus.cc:729] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 51eee958aa8948b1a46aa1641638b9ec, State: Running, Role: LEADER
I20260810 18:50:38.840435 15878 consensus_queue.cc:260] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } }
I20260810 18:50:38.841046 15752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b96127e94b74666aa7a5d280228ba7d" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
I20260810 18:50:38.841426 15878 raft_consensus.cc:3037] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:38.844772 15613 raft_consensus.cc:1226] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [term 2 FOLLOWER]: Rejecting Update request from peer 51eee958aa8948b1a46aa1641638b9ec for earlier term 1. Current term is 2. Ops: []
I20260810 18:50:38.848055 15878 raft_consensus.cc:2454] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffe600335b884cae8bdd12c2ef74e670 in term 2.
I20260810 18:50:38.848975 15537 leader_election.cc:302] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51eee958aa8948b1a46aa1641638b9ec, ffe600335b884cae8bdd12c2ef74e670; no voters: 2d28250046da4d688d76be4ae6e4c826
I20260810 18:50:38.849728 15979 raft_consensus.cc:2781] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:50:38.850194 15979 raft_consensus.cc:686] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [term 2 LEADER]: Becoming Leader. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Running, Role: LEADER
I20260810 18:50:38.850898 15979 consensus_queue.cc:237] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } }
I20260810 18:50:38.857443 15418 catalog_manager.cc:5495] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 reported cstate change: term changed from 1 to 2, leader changed from 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195) to ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193). New cstate: current_term: 2 leader_uuid: "ffe600335b884cae8bdd12c2ef74e670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:38.932917 16050 consensus_queue.cc:1035] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:50:38.939793 16043 raft_consensus.cc:979] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec: : Instructing follower ffe600335b884cae8bdd12c2ef74e670 to start an election
I20260810 18:50:38.940243 16043 raft_consensus.cc:1067] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 1 LEADER]: Signalling peer ffe600335b884cae8bdd12c2ef74e670 to start an election
I20260810 18:50:38.942850 15612 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "354460a175b24dcd873c370589a0f7d5"
dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
from {username='slave'} at 127.0.0.1:60318
I20260810 18:50:38.943852 15612 raft_consensus.cc:482] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:38.944242 15612 raft_consensus.cc:3037] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:38.944494 15967 consensus_queue.cc:1035] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:50:38.951349 15612 raft_consensus.cc:504] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.953289 15612 leader_election.cc:288] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 2 election: Requested vote from peers 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:38.954348 15878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354460a175b24dcd873c370589a0f7d5" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
I20260810 18:50:38.954833 15878 raft_consensus.cc:3032] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:50:38.955494 15878 raft_consensus.cc:729] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 51eee958aa8948b1a46aa1641638b9ec, State: Running, Role: LEADER
I20260810 18:50:38.955489 15753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354460a175b24dcd873c370589a0f7d5" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
I20260810 18:50:38.956278 15753 raft_consensus.cc:3037] T 354460a175b24dcd873c370589a0f7d5 P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:38.956471 15878 consensus_queue.cc:260] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.957765 15878 raft_consensus.cc:3037] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:38.964258 15878 raft_consensus.cc:2454] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffe600335b884cae8bdd12c2ef74e670 in term 2.
I20260810 18:50:38.964506 15753 raft_consensus.cc:2454] T 354460a175b24dcd873c370589a0f7d5 P 2d28250046da4d688d76be4ae6e4c826 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffe600335b884cae8bdd12c2ef74e670 in term 2.
I20260810 18:50:38.965322 15537 leader_election.cc:302] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51eee958aa8948b1a46aa1641638b9ec, ffe600335b884cae8bdd12c2ef74e670; no voters:
I20260810 18:50:38.966100 15979 raft_consensus.cc:2781] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:50:38.966579 15979 raft_consensus.cc:686] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [term 2 LEADER]: Becoming Leader. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Running, Role: LEADER
I20260810 18:50:38.967410 15979 consensus_queue.cc:237] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.972992 15417 catalog_manager.cc:5495] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 reported cstate change: term changed from 1 to 2, leader changed from 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195) to ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193). New cstate: current_term: 2 leader_uuid: "ffe600335b884cae8bdd12c2ef74e670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:38.981552 15753 raft_consensus.cc:1226] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [term 2 LEADER]: Rejecting Update request from peer 51eee958aa8948b1a46aa1641638b9ec for earlier term 1. Current term is 2. Ops: []
I20260810 18:50:38.982688 16043 consensus_queue.cc:1046] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:50:38.983392 16043 consensus_queue.cc:1225] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [LEADER]: Peer 2d28250046da4d688d76be4ae6e4c826 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20260810 18:50:38.983842 15967 raft_consensus.cc:3032] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:50:38.984095 15967 raft_consensus.cc:729] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 51eee958aa8948b1a46aa1641638b9ec, State: Running, Role: LEADER
I20260810 18:50:38.984664 15967 consensus_queue.cc:260] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:38.985637 15967 raft_consensus.cc:3037] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:39.045373 16004 raft_consensus.cc:979] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826: : Instructing follower ffe600335b884cae8bdd12c2ef74e670 to start an election
I20260810 18:50:39.046021 16004 raft_consensus.cc:1067] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [term 1 LEADER]: Signalling peer ffe600335b884cae8bdd12c2ef74e670 to start an election
I20260810 18:50:39.099193 16004 consensus_queue.cc:1035] T cd4103b9a0414848b8ad7e116bf4ded8 P 2d28250046da4d688d76be4ae6e4c826 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:50:39.164301 15612 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f6b952757814de89b38a47f3942ee7e"
dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
from {username='slave'} at 127.0.0.1:60324
I20260810 18:50:39.164960 15612 raft_consensus.cc:482] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:39.165320 15612 raft_consensus.cc:3037] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:39.165592 15613 raft_consensus.cc:1261] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 2 FOLLOWER]: Refusing update from remote peer 2d28250046da4d688d76be4ae6e4c826: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260810 18:50:39.172284 15612 raft_consensus.cc:504] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:39.175211 15612 leader_election.cc:288] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 2 election: Requested vote from peers 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:39.205577 15879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f6b952757814de89b38a47f3942ee7e" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
I20260810 18:50:39.206259 15879 raft_consensus.cc:3037] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:39.215575 15879 raft_consensus.cc:2454] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffe600335b884cae8bdd12c2ef74e670 in term 2.
I20260810 18:50:39.217012 15537 leader_election.cc:302] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51eee958aa8948b1a46aa1641638b9ec, ffe600335b884cae8bdd12c2ef74e670; no voters:
I20260810 18:50:39.233880 15979 raft_consensus.cc:2781] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:50:39.234445 15979 raft_consensus.cc:686] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670 [term 2 LEADER]: Becoming Leader. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Running, Role: LEADER
I20260810 18:50:39.235447 15979 consensus_queue.cc:237] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:39.237857 16003 consensus_queue.cc:1035] T cd4103b9a0414848b8ad7e116bf4ded8 P 2d28250046da4d688d76be4ae6e4c826 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:50:39.239216 16003 consensus_queue.cc:1035] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:50:39.240447 16003 raft_consensus.cc:979] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826: : Instructing follower ffe600335b884cae8bdd12c2ef74e670 to start an election
I20260810 18:50:39.240805 16003 raft_consensus.cc:1067] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [term 1 LEADER]: Signalling peer ffe600335b884cae8bdd12c2ef74e670 to start an election
I20260810 18:50:39.244491 15611 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "20f91c7cf7b14e6ab5215e47458b4f1b"
dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
from {username='slave'} at 127.0.0.1:60324
I20260810 18:50:39.245066 15611 raft_consensus.cc:482] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:39.245426 15611 raft_consensus.cc:3037] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:39.251220 16003 raft_consensus.cc:979] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826: : Instructing follower ffe600335b884cae8bdd12c2ef74e670 to start an election
I20260810 18:50:39.251681 16060 raft_consensus.cc:1067] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [term 2 LEADER]: Signalling peer ffe600335b884cae8bdd12c2ef74e670 to start an election
I20260810 18:50:39.253165 15613 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e7e5673440be4239a048129ffad7a53c"
dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
from {username='slave'} at 127.0.0.1:60324
I20260810 18:50:39.255308 15878 raft_consensus.cc:1261] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 2 FOLLOWER]: Refusing update from remote peer 2d28250046da4d688d76be4ae6e4c826: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260810 18:50:39.255728 15878 pending_rounds.cc:76] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec: Aborting all ops after (but not including) 0
I20260810 18:50:39.255976 15878 pending_rounds.cc:97] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260810 18:50:39.256300 15878 raft_consensus.cc:2866] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:50:39.257356 15752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f6b952757814de89b38a47f3942ee7e" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
I20260810 18:50:39.259310 15979 raft_consensus.cc:979] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670: : Instructing follower 2d28250046da4d688d76be4ae6e4c826 to start an election
I20260810 18:50:39.259802 15979 raft_consensus.cc:1067] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [term 1 LEADER]: Signalling peer 2d28250046da4d688d76be4ae6e4c826 to start an election
I20260810 18:50:39.260324 15613 raft_consensus.cc:482] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:39.260713 15613 raft_consensus.cc:3037] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:39.261035 15752 raft_consensus.cc:3032] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:50:39.261405 15752 raft_consensus.cc:729] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2d28250046da4d688d76be4ae6e4c826, State: Running, Role: LEADER
I20260810 18:50:39.267249 16060 consensus_queue.cc:1035] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:50:39.268504 15750 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9764b4e886a24c9db19a64075920a794"
dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
from {username='slave'} at 127.0.0.1:59498
I20260810 18:50:39.268941 15750 raft_consensus.cc:482] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:39.269271 15750 raft_consensus.cc:3037] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:39.269659 15878 raft_consensus.cc:1226] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec [term 2 FOLLOWER]: Rejecting Update request from peer 2d28250046da4d688d76be4ae6e4c826 for earlier term 1. Current term is 2. Ops: []
I20260810 18:50:39.266113 15752 consensus_queue.cc:260] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:39.277338 16006 raft_consensus.cc:979] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670: : Instructing follower 2d28250046da4d688d76be4ae6e4c826 to start an election
I20260810 18:50:39.277798 16006 raft_consensus.cc:1067] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [term 1 LEADER]: Signalling peer 2d28250046da4d688d76be4ae6e4c826 to start an election
I20260810 18:50:39.277774 15752 raft_consensus.cc:3037] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:39.279489 15753 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8e61e9bbff154604bce6712d94c73e90"
dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
from {username='slave'} at 127.0.0.1:59498
I20260810 18:50:39.280040 15753 raft_consensus.cc:482] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:39.280404 15753 raft_consensus.cc:3037] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:39.281932 15613 raft_consensus.cc:504] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:39.285341 15417 catalog_manager.cc:5495] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670 reported cstate change: term changed from 1 to 2, leader changed from 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194) to ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193). New cstate: current_term: 2 leader_uuid: "ffe600335b884cae8bdd12c2ef74e670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:39.287740 15753 raft_consensus.cc:504] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:39.290475 15752 raft_consensus.cc:2454] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffe600335b884cae8bdd12c2ef74e670 in term 2.
I20260810 18:50:39.292574 15750 raft_consensus.cc:504] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:39.292910 15878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e5673440be4239a048129ffad7a53c" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
I20260810 18:50:39.293419 15745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e5673440be4239a048129ffad7a53c" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
I20260810 18:50:39.293975 15745 raft_consensus.cc:3032] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:50:39.294039 15613 leader_election.cc:288] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 3 election: Requested vote from peers 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:39.294360 15745 raft_consensus.cc:729] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2d28250046da4d688d76be4ae6e4c826, State: Running, Role: LEADER
I20260810 18:50:39.295162 15745 consensus_queue.cc:260] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:39.296020 15879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9764b4e886a24c9db19a64075920a794" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
I20260810 18:50:39.296259 15745 raft_consensus.cc:3037] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:39.296192 15611 raft_consensus.cc:504] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:39.296725 15879 raft_consensus.cc:3037] T 9764b4e886a24c9db19a64075920a794 P 51eee958aa8948b1a46aa1641638b9ec [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:39.298211 15611 leader_election.cc:288] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 2 election: Requested vote from peers 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:39.299378 15876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f91c7cf7b14e6ab5215e47458b4f1b" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
I20260810 18:50:39.293571 15878 raft_consensus.cc:3037] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:39.299999 15876 raft_consensus.cc:3037] T 20f91c7cf7b14e6ab5215e47458b4f1b P 51eee958aa8948b1a46aa1641638b9ec [term 1 FOLLOWER]: Advancing to term 2
W20260810 18:50:39.280382 16060 consensus_queue.cc:1159] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "51eee958aa8948b1a46aa1641638b9ec" responder_term: 2 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 2d28250046da4d688d76be4ae6e4c826 for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260810 18:50:39.302961 15750 leader_election.cc:288] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 2 election: Requested vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:39.302881 15743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f91c7cf7b14e6ab5215e47458b4f1b" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
I20260810 18:50:39.303577 15743 raft_consensus.cc:3032] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:50:39.303879 15743 raft_consensus.cc:729] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2d28250046da4d688d76be4ae6e4c826, State: Running, Role: LEADER
I20260810 18:50:39.304490 15743 consensus_queue.cc:260] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:39.305220 15743 raft_consensus.cc:3037] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:39.312219 15876 raft_consensus.cc:2454] T 20f91c7cf7b14e6ab5215e47458b4f1b P 51eee958aa8948b1a46aa1641638b9ec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffe600335b884cae8bdd12c2ef74e670 in term 2.
I20260810 18:50:39.313915 15878 raft_consensus.cc:2454] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffe600335b884cae8bdd12c2ef74e670 in term 3.
I20260810 18:50:39.314569 15874 raft_consensus.cc:1261] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [term 2 FOLLOWER]: Refusing update from remote peer ffe600335b884cae8bdd12c2ef74e670: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:39.315285 15743 raft_consensus.cc:2454] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffe600335b884cae8bdd12c2ef74e670 in term 2.
I20260810 18:50:39.315692 15879 raft_consensus.cc:2454] T 9764b4e886a24c9db19a64075920a794 P 51eee958aa8948b1a46aa1641638b9ec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d28250046da4d688d76be4ae6e4c826 in term 2.
I20260810 18:50:39.316211 15537 leader_election.cc:302] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51eee958aa8948b1a46aa1641638b9ec, ffe600335b884cae8bdd12c2ef74e670; no voters:
I20260810 18:50:39.316911 15673 leader_election.cc:302] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d28250046da4d688d76be4ae6e4c826, 51eee958aa8948b1a46aa1641638b9ec; no voters:
I20260810 18:50:39.316987 16006 raft_consensus.cc:2781] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:50:39.317123 15537 leader_election.cc:302] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51eee958aa8948b1a46aa1641638b9ec, ffe600335b884cae8bdd12c2ef74e670; no voters:
I20260810 18:50:39.317675 16006 raft_consensus.cc:686] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670 [term 2 LEADER]: Becoming Leader. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Running, Role: LEADER
I20260810 18:50:39.317832 16060 raft_consensus.cc:2781] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:50:39.318050 16077 raft_consensus.cc:2781] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:50:39.318534 16077 raft_consensus.cc:686] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 3 LEADER]: Becoming Leader. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Running, Role: LEADER
I20260810 18:50:39.318974 15745 raft_consensus.cc:2454] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffe600335b884cae8bdd12c2ef74e670 in term 3.
I20260810 18:50:39.319217 16060 raft_consensus.cc:686] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826 [term 2 LEADER]: Becoming Leader. State: Replica: 2d28250046da4d688d76be4ae6e4c826, State: Running, Role: LEADER
I20260810 18:50:39.319355 16077 consensus_queue.cc:237] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:39.318552 16006 consensus_queue.cc:237] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:39.320041 15612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e61e9bbff154604bce6712d94c73e90" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
I20260810 18:50:39.320700 15612 raft_consensus.cc:3032] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:50:39.321012 15612 raft_consensus.cc:729] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Running, Role: LEADER
I20260810 18:50:39.320480 16060 consensus_queue.cc:237] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:39.321762 15612 consensus_queue.cc:260] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:39.322909 15612 raft_consensus.cc:3037] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:39.324396 15611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9764b4e886a24c9db19a64075920a794" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
I20260810 18:50:39.324898 15611 raft_consensus.cc:3032] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:50:39.325150 15611 raft_consensus.cc:729] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Running, Role: LEADER
I20260810 18:50:39.325824 15611 consensus_queue.cc:260] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:39.327056 15611 raft_consensus.cc:3037] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:39.327908 15745 raft_consensus.cc:3037] T 8b96127e94b74666aa7a5d280228ba7d P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:39.330415 15612 raft_consensus.cc:2454] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d28250046da4d688d76be4ae6e4c826 in term 2.
I20260810 18:50:39.330103 15417 catalog_manager.cc:5495] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670 reported cstate change: term changed from 1 to 2, leader changed from 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194) to ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193). New cstate: current_term: 2 leader_uuid: "ffe600335b884cae8bdd12c2ef74e670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:39.331243 16006 consensus_queue.cc:1035] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:50:39.331554 15675 leader_election.cc:302] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d28250046da4d688d76be4ae6e4c826, ffe600335b884cae8bdd12c2ef74e670; no voters:
I20260810 18:50:39.332381 15977 raft_consensus.cc:2781] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:50:39.332031 15417 catalog_manager.cc:5495] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 reported cstate change: term changed from 2 to 3, leader changed from 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194) to ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193). New cstate: current_term: 3 leader_uuid: "ffe600335b884cae8bdd12c2ef74e670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:39.332839 15977 raft_consensus.cc:686] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [term 2 LEADER]: Becoming Leader. State: Replica: 2d28250046da4d688d76be4ae6e4c826, State: Running, Role: LEADER
I20260810 18:50:39.333523 15611 raft_consensus.cc:2454] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d28250046da4d688d76be4ae6e4c826 in term 2.
I20260810 18:50:39.333724 15977 consensus_queue.cc:237] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:39.334811 15753 leader_election.cc:288] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 2 election: Requested vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:39.339401 15745 raft_consensus.cc:1261] T 8b96127e94b74666aa7a5d280228ba7d P 2d28250046da4d688d76be4ae6e4c826 [term 2 FOLLOWER]: Refusing update from remote peer ffe600335b884cae8bdd12c2ef74e670: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:39.340824 15979 consensus_queue.cc:1035] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:50:39.347123 15415 catalog_manager.cc:5495] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826 reported cstate change: term changed from 1 to 2, leader changed from ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193) to 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194). New cstate: current_term: 2 leader_uuid: "2d28250046da4d688d76be4ae6e4c826" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:39.350338 15879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e61e9bbff154604bce6712d94c73e90" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
I20260810 18:50:39.351106 15879 raft_consensus.cc:3037] T 8e61e9bbff154604bce6712d94c73e90 P 51eee958aa8948b1a46aa1641638b9ec [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:39.357793 15879 raft_consensus.cc:2454] T 8e61e9bbff154604bce6712d94c73e90 P 51eee958aa8948b1a46aa1641638b9ec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d28250046da4d688d76be4ae6e4c826 in term 2.
I20260810 18:50:39.365648 15415 catalog_manager.cc:5495] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 reported cstate change: term changed from 1 to 2, leader changed from ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193) to 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194). New cstate: current_term: 2 leader_uuid: "2d28250046da4d688d76be4ae6e4c826" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:39.441947 15879 raft_consensus.cc:1261] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 2 FOLLOWER]: Refusing update from remote peer ffe600335b884cae8bdd12c2ef74e670: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:39.443300 16077 consensus_queue.cc:1035] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:50:39.450805 15745 raft_consensus.cc:1261] T 354460a175b24dcd873c370589a0f7d5 P 2d28250046da4d688d76be4ae6e4c826 [term 2 FOLLOWER]: Refusing update from remote peer ffe600335b884cae8bdd12c2ef74e670: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:39.452231 16006 consensus_queue.cc:1035] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:50:39.713080 15611 raft_consensus.cc:1261] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [term 2 FOLLOWER]: Refusing update from remote peer 2d28250046da4d688d76be4ae6e4c826: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:39.714216 15977 consensus_queue.cc:1035] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:50:39.722453 15879 raft_consensus.cc:1261] T 8e61e9bbff154604bce6712d94c73e90 P 51eee958aa8948b1a46aa1641638b9ec [term 2 FOLLOWER]: Refusing update from remote peer 2d28250046da4d688d76be4ae6e4c826: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:39.723822 15977 consensus_queue.cc:1035] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:50:39.730017 15876 raft_consensus.cc:1261] T 9764b4e886a24c9db19a64075920a794 P 51eee958aa8948b1a46aa1641638b9ec [term 2 FOLLOWER]: Refusing update from remote peer 2d28250046da4d688d76be4ae6e4c826: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:39.731280 15977 consensus_queue.cc:1035] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:50:39.739830 15611 raft_consensus.cc:1261] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [term 2 FOLLOWER]: Refusing update from remote peer 2d28250046da4d688d76be4ae6e4c826: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:39.741297 16003 consensus_queue.cc:1035] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:50:39.777758 15879 raft_consensus.cc:1261] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec [term 2 FOLLOWER]: Refusing update from remote peer ffe600335b884cae8bdd12c2ef74e670: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:39.778852 16006 consensus_queue.cc:1035] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:50:39.783282 15745 raft_consensus.cc:1261] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [term 2 FOLLOWER]: Refusing update from remote peer ffe600335b884cae8bdd12c2ef74e670: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:39.784955 16006 consensus_queue.cc:1035] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:50:39.829236 15745 raft_consensus.cc:1261] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [term 3 FOLLOWER]: Refusing update from remote peer ffe600335b884cae8bdd12c2ef74e670: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260810 18:50:39.830542 16077 consensus_queue.cc:1035] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:50:39.839450 15876 raft_consensus.cc:1261] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 3 FOLLOWER]: Refusing update from remote peer ffe600335b884cae8bdd12c2ef74e670: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260810 18:50:39.841259 16123 consensus_queue.cc:1035] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:50:39.891868 15745 raft_consensus.cc:1261] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [term 2 FOLLOWER]: Refusing update from remote peer ffe600335b884cae8bdd12c2ef74e670: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:39.893110 16123 consensus_queue.cc:1035] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:50:39.901448 15879 raft_consensus.cc:1261] T 20f91c7cf7b14e6ab5215e47458b4f1b P 51eee958aa8948b1a46aa1641638b9ec [term 2 FOLLOWER]: Refusing update from remote peer ffe600335b884cae8bdd12c2ef74e670: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:39.903044 16006 consensus_queue.cc:1035] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260810 18:50:40.158062 15647 debug-util.cc:398] Leaking SignalData structure 0x7b08001bccc0 after lost signal to thread 14470
W20260810 18:50:40.158869 15647 debug-util.cc:398] Leaking SignalData structure 0x7b08001bc940 after lost signal to thread 15490
W20260810 18:50:40.159795 15647 debug-util.cc:398] Leaking SignalData structure 0x7b08001d98e0 after lost signal to thread 15650
W20260810 18:50:40.160538 15647 debug-util.cc:398] Leaking SignalData structure 0x7b08000cae60 after lost signal to thread 15789
W20260810 18:50:40.161262 15647 debug-util.cc:398] Leaking SignalData structure 0x7b08001e2980 after lost signal to thread 15932
I20260810 18:50:41.912448 14467 auto_leader_rebalancer-test.cc:339] The leader distribution is
I20260810 18:50:41.913396 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:41.950287 15612 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "354460a175b24dcd873c370589a0f7d5"
dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
mode: GRACEFUL
new_leader_uuid: "51eee958aa8948b1a46aa1641638b9ec"
from {username='slave'} at 127.0.0.1:60338
I20260810 18:50:41.951150 15612 raft_consensus.cc:595] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [term 2 LEADER]: Received request to transfer leadership to TS 51eee958aa8948b1a46aa1641638b9ec
I20260810 18:50:41.953812 15612 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e7e5673440be4239a048129ffad7a53c"
dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
mode: GRACEFUL
new_leader_uuid: "51eee958aa8948b1a46aa1641638b9ec"
from {username='slave'} at 127.0.0.1:60338
I20260810 18:50:41.954418 15612 raft_consensus.cc:595] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 3 LEADER]: Received request to transfer leadership to TS 51eee958aa8948b1a46aa1641638b9ec
I20260810 18:50:41.955704 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260810 18:50:41.956138 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:41.977425 16077 raft_consensus.cc:979] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670: : Instructing follower 51eee958aa8948b1a46aa1641638b9ec to start an election
I20260810 18:50:41.978055 16077 raft_consensus.cc:1067] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 3 LEADER]: Signalling peer 51eee958aa8948b1a46aa1641638b9ec to start an election
I20260810 18:50:41.980276 15879 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e7e5673440be4239a048129ffad7a53c"
dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
from {username='slave'} at 127.0.0.1:35178
I20260810 18:50:41.980911 15879 raft_consensus.cc:482] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:41.981356 15879 raft_consensus.cc:3037] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:50:41.987315 15879 raft_consensus.cc:504] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:41.989041 15879 leader_election.cc:288] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 4 election: Requested vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629)
I20260810 18:50:41.990106 15745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e5673440be4239a048129ffad7a53c" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
I20260810 18:50:41.990095 15612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e5673440be4239a048129ffad7a53c" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
I20260810 18:50:41.990918 15745 raft_consensus.cc:3037] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:50:41.991017 15612 raft_consensus.cc:3032] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:50:41.991452 15612 raft_consensus.cc:729] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Running, Role: LEADER
I20260810 18:50:41.992265 15612 consensus_queue.cc:260] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:41.993618 15612 raft_consensus.cc:3037] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:50:41.998119 15745 raft_consensus.cc:2454] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51eee958aa8948b1a46aa1641638b9ec in term 4.
I20260810 18:50:41.999469 15816 leader_election.cc:302] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d28250046da4d688d76be4ae6e4c826, 51eee958aa8948b1a46aa1641638b9ec; no voters:
I20260810 18:50:42.000355 16134 raft_consensus.cc:2781] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:50:42.000841 15612 raft_consensus.cc:2454] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51eee958aa8948b1a46aa1641638b9ec in term 4.
I20260810 18:50:42.001755 16134 raft_consensus.cc:686] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 4 LEADER]: Becoming Leader. State: Replica: 51eee958aa8948b1a46aa1641638b9ec, State: Running, Role: LEADER
I20260810 18:50:42.002650 16134 consensus_queue.cc:237] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:42.010965 15417 catalog_manager.cc:5495] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec reported cstate change: term changed from 3 to 4, leader changed from ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193) to 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195). New cstate: current_term: 4 leader_uuid: "51eee958aa8948b1a46aa1641638b9ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } health_report { overall_health: HEALTHY } } }
I20260810 18:50:42.441920 15612 raft_consensus.cc:1261] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 4 FOLLOWER]: Refusing update from remote peer 51eee958aa8948b1a46aa1641638b9ec: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260810 18:50:42.443694 16144 consensus_queue.cc:1035] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:50:42.457435 15745 raft_consensus.cc:1261] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [term 4 FOLLOWER]: Refusing update from remote peer 51eee958aa8948b1a46aa1641638b9ec: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260810 18:50:42.461229 16144 consensus_queue.cc:1035] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:50:42.470742 16077 raft_consensus.cc:979] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670: : Instructing follower 51eee958aa8948b1a46aa1641638b9ec to start an election
I20260810 18:50:42.471385 16077 raft_consensus.cc:1067] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [term 2 LEADER]: Signalling peer 51eee958aa8948b1a46aa1641638b9ec to start an election
I20260810 18:50:42.478556 15879 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "354460a175b24dcd873c370589a0f7d5"
dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
from {username='slave'} at 127.0.0.1:35178
I20260810 18:50:42.479176 15879 raft_consensus.cc:482] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:42.479563 15879 raft_consensus.cc:3037] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:42.488236 15879 raft_consensus.cc:504] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:42.490439 15879 leader_election.cc:288] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 3 election: Requested vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629)
I20260810 18:50:42.491238 15745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354460a175b24dcd873c370589a0f7d5" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
I20260810 18:50:42.491946 15745 raft_consensus.cc:3037] T 354460a175b24dcd873c370589a0f7d5 P 2d28250046da4d688d76be4ae6e4c826 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:42.496935 15612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354460a175b24dcd873c370589a0f7d5" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
I20260810 18:50:42.497582 15612 raft_consensus.cc:3032] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:50:42.497934 15612 raft_consensus.cc:729] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Running, Role: LEADER
I20260810 18:50:42.498824 15612 consensus_queue.cc:260] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:42.499354 15745 raft_consensus.cc:2454] T 354460a175b24dcd873c370589a0f7d5 P 2d28250046da4d688d76be4ae6e4c826 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51eee958aa8948b1a46aa1641638b9ec in term 3.
I20260810 18:50:42.500284 15612 raft_consensus.cc:3037] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:42.500420 15816 leader_election.cc:302] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d28250046da4d688d76be4ae6e4c826, 51eee958aa8948b1a46aa1641638b9ec; no voters:
I20260810 18:50:42.501305 16134 raft_consensus.cc:2781] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:50:42.501806 16134 raft_consensus.cc:686] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 3 LEADER]: Becoming Leader. State: Replica: 51eee958aa8948b1a46aa1641638b9ec, State: Running, Role: LEADER
I20260810 18:50:42.502633 16134 consensus_queue.cc:237] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:42.505963 15612 raft_consensus.cc:2454] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51eee958aa8948b1a46aa1641638b9ec in term 3.
I20260810 18:50:42.511536 15417 catalog_manager.cc:5495] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec reported cstate change: term changed from 2 to 3, leader changed from ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193) to 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195). New cstate: current_term: 3 leader_uuid: "51eee958aa8948b1a46aa1641638b9ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } health_report { overall_health: HEALTHY } } }
I20260810 18:50:42.895654 15612 raft_consensus.cc:1261] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [term 3 FOLLOWER]: Refusing update from remote peer 51eee958aa8948b1a46aa1641638b9ec: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:42.897348 16134 consensus_queue.cc:1035] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:50:42.905896 15745 raft_consensus.cc:1261] T 354460a175b24dcd873c370589a0f7d5 P 2d28250046da4d688d76be4ae6e4c826 [term 3 FOLLOWER]: Refusing update from remote peer 51eee958aa8948b1a46aa1641638b9ec: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:42.907517 16144 consensus_queue.cc:1035] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:50:42.956995 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:42.962260 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:42.962620 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:43.963867 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:43.970629 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:43.971081 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:44.972002 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:44.977147 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:44.977569 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:45.978379 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:45.984606 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:45.985026 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:46.985915 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:46.991138 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:46.991483 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:47.992206 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:47.997219 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:47.997550 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:48.998318 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:49.003787 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:49.004251 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:50.005045 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:50.010068 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:50.010468 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:51.011399 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:51.016523 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:51.016949 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:52.046538 14467 auto_leader_rebalancer-test.cc:358] The leader distribution is
I20260810 18:50:52.058441 15876 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "354460a175b24dcd873c370589a0f7d5"
dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
mode: GRACEFUL
new_leader_uuid: "ffe600335b884cae8bdd12c2ef74e670"
from {username='slave'} at 127.0.0.1:35188
I20260810 18:50:52.059201 15876 raft_consensus.cc:595] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 3 LEADER]: Received request to transfer leadership to TS ffe600335b884cae8bdd12c2ef74e670
I20260810 18:50:52.063828 15876 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e7e5673440be4239a048129ffad7a53c"
dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
mode: GRACEFUL
new_leader_uuid: "ffe600335b884cae8bdd12c2ef74e670"
from {username='slave'} at 127.0.0.1:35188
I20260810 18:50:52.064321 15876 raft_consensus.cc:595] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 4 LEADER]: Received request to transfer leadership to TS ffe600335b884cae8bdd12c2ef74e670
I20260810 18:50:52.068919 15753 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2813a19d92094a648be20c4c20f713fa"
dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
mode: GRACEFUL
new_leader_uuid: "ffe600335b884cae8bdd12c2ef74e670"
from {username='slave'} at 127.0.0.1:59512
I20260810 18:50:52.069416 15753 raft_consensus.cc:595] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [term 1 LEADER]: Received request to transfer leadership to TS ffe600335b884cae8bdd12c2ef74e670
I20260810 18:50:52.073942 15753 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8e61e9bbff154604bce6712d94c73e90"
dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
mode: GRACEFUL
new_leader_uuid: "ffe600335b884cae8bdd12c2ef74e670"
from {username='slave'} at 127.0.0.1:59512
I20260810 18:50:52.074514 15753 raft_consensus.cc:595] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [term 2 LEADER]: Received request to transfer leadership to TS ffe600335b884cae8bdd12c2ef74e670
I20260810 18:50:52.078737 15753 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9764b4e886a24c9db19a64075920a794"
dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
mode: GRACEFUL
new_leader_uuid: "ffe600335b884cae8bdd12c2ef74e670"
from {username='slave'} at 127.0.0.1:59512
I20260810 18:50:52.079210 15753 raft_consensus.cc:595] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826 [term 2 LEADER]: Received request to transfer leadership to TS ffe600335b884cae8bdd12c2ef74e670
I20260810 18:50:52.162619 16178 raft_consensus.cc:979] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec: : Instructing follower ffe600335b884cae8bdd12c2ef74e670 to start an election
I20260810 18:50:52.163187 16134 raft_consensus.cc:1067] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 3 LEADER]: Signalling peer ffe600335b884cae8bdd12c2ef74e670 to start an election
I20260810 18:50:52.164700 15612 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "354460a175b24dcd873c370589a0f7d5"
dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
from {username='slave'} at 127.0.0.1:60318
I20260810 18:50:52.165201 15612 raft_consensus.cc:482] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:52.165594 15612 raft_consensus.cc:3037] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:50:52.170854 15612 raft_consensus.cc:504] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:52.172523 15612 leader_election.cc:288] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 4 election: Requested vote from peers 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:52.173420 15753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354460a175b24dcd873c370589a0f7d5" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
I20260810 18:50:52.173974 15876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354460a175b24dcd873c370589a0f7d5" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
I20260810 18:50:52.174117 15753 raft_consensus.cc:3037] T 354460a175b24dcd873c370589a0f7d5 P 2d28250046da4d688d76be4ae6e4c826 [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:50:52.174544 15876 raft_consensus.cc:3032] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:50:52.174868 15876 raft_consensus.cc:729] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 51eee958aa8948b1a46aa1641638b9ec, State: Running, Role: LEADER
I20260810 18:50:52.175652 15876 consensus_queue.cc:260] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:52.176824 15876 raft_consensus.cc:3037] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:50:52.180025 15753 raft_consensus.cc:2454] T 354460a175b24dcd873c370589a0f7d5 P 2d28250046da4d688d76be4ae6e4c826 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffe600335b884cae8bdd12c2ef74e670 in term 4.
I20260810 18:50:52.181411 15537 leader_election.cc:302] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d28250046da4d688d76be4ae6e4c826, ffe600335b884cae8bdd12c2ef74e670; no voters:
I20260810 18:50:52.182339 16175 raft_consensus.cc:2781] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:50:52.183370 15876 raft_consensus.cc:2454] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffe600335b884cae8bdd12c2ef74e670 in term 4.
I20260810 18:50:52.183521 16175 raft_consensus.cc:686] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [term 4 LEADER]: Becoming Leader. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Running, Role: LEADER
I20260810 18:50:52.185331 16175 consensus_queue.cc:237] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:52.191859 15415 catalog_manager.cc:5495] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 reported cstate change: term changed from 3 to 4, leader changed from 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195) to ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193). New cstate: current_term: 4 leader_uuid: "ffe600335b884cae8bdd12c2ef74e670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:52.303769 16191 raft_consensus.cc:979] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826: : Instructing follower ffe600335b884cae8bdd12c2ef74e670 to start an election
I20260810 18:50:52.304337 16173 raft_consensus.cc:1067] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [term 1 LEADER]: Signalling peer ffe600335b884cae8bdd12c2ef74e670 to start an election
I20260810 18:50:52.305814 15612 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2813a19d92094a648be20c4c20f713fa"
dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
from {username='slave'} at 127.0.0.1:60324
I20260810 18:50:52.306301 15612 raft_consensus.cc:482] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:52.306610 15612 raft_consensus.cc:3037] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:52.311511 15612 raft_consensus.cc:504] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:52.312885 15612 leader_election.cc:288] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 2 election: Requested vote from peers 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:52.313815 15753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2813a19d92094a648be20c4c20f713fa" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
I20260810 18:50:52.314009 15876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2813a19d92094a648be20c4c20f713fa" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
I20260810 18:50:52.314327 15753 raft_consensus.cc:3032] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:50:52.314600 15876 raft_consensus.cc:3037] T 2813a19d92094a648be20c4c20f713fa P 51eee958aa8948b1a46aa1641638b9ec [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:52.314638 15753 raft_consensus.cc:729] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2d28250046da4d688d76be4ae6e4c826, State: Running, Role: LEADER
I20260810 18:50:52.315636 15753 consensus_queue.cc:260] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:52.316671 15753 raft_consensus.cc:3037] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:52.319911 15876 raft_consensus.cc:2454] T 2813a19d92094a648be20c4c20f713fa P 51eee958aa8948b1a46aa1641638b9ec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffe600335b884cae8bdd12c2ef74e670 in term 2.
I20260810 18:50:52.321044 15537 leader_election.cc:302] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51eee958aa8948b1a46aa1641638b9ec, ffe600335b884cae8bdd12c2ef74e670; no voters:
I20260810 18:50:52.321558 15753 raft_consensus.cc:2454] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffe600335b884cae8bdd12c2ef74e670 in term 2.
I20260810 18:50:52.321839 16175 raft_consensus.cc:2781] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:50:52.322324 16175 raft_consensus.cc:686] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670 [term 2 LEADER]: Becoming Leader. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Running, Role: LEADER
I20260810 18:50:52.323216 16175 consensus_queue.cc:237] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:52.329290 15415 catalog_manager.cc:5495] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670 reported cstate change: term changed from 1 to 2, leader changed from 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194) to ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193). New cstate: current_term: 2 leader_uuid: "ffe600335b884cae8bdd12c2ef74e670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:52.356616 16178 raft_consensus.cc:979] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec: : Instructing follower ffe600335b884cae8bdd12c2ef74e670 to start an election
I20260810 18:50:52.357079 16134 raft_consensus.cc:1067] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 4 LEADER]: Signalling peer ffe600335b884cae8bdd12c2ef74e670 to start an election
I20260810 18:50:52.358284 15612 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e7e5673440be4239a048129ffad7a53c"
dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
from {username='slave'} at 127.0.0.1:60318
I20260810 18:50:52.358734 15612 raft_consensus.cc:482] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:52.358966 15612 raft_consensus.cc:3037] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:50:52.363620 15612 raft_consensus.cc:504] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:52.364883 15612 leader_election.cc:288] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 5 election: Requested vote from peers 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:52.365875 15753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e5673440be4239a048129ffad7a53c" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
I20260810 18:50:52.366161 15876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e5673440be4239a048129ffad7a53c" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
I20260810 18:50:52.366521 15753 raft_consensus.cc:3037] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:50:52.366694 15876 raft_consensus.cc:3032] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 4 LEADER]: Stepping down as leader of term 4
I20260810 18:50:52.367036 15876 raft_consensus.cc:729] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 51eee958aa8948b1a46aa1641638b9ec, State: Running, Role: LEADER
I20260810 18:50:52.367697 15876 consensus_queue.cc:260] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:52.368620 15876 raft_consensus.cc:3037] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:50:52.372714 15753 raft_consensus.cc:2454] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffe600335b884cae8bdd12c2ef74e670 in term 5.
I20260810 18:50:52.373335 15876 raft_consensus.cc:2454] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffe600335b884cae8bdd12c2ef74e670 in term 5.
I20260810 18:50:52.373860 15537 leader_election.cc:302] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d28250046da4d688d76be4ae6e4c826, ffe600335b884cae8bdd12c2ef74e670; no voters:
I20260810 18:50:52.374681 16175 raft_consensus.cc:2781] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:50:52.375180 16175 raft_consensus.cc:686] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 5 LEADER]: Becoming Leader. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Running, Role: LEADER
I20260810 18:50:52.375936 16175 consensus_queue.cc:237] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:52.382241 15415 catalog_manager.cc:5495] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 reported cstate change: term changed from 4 to 5, leader changed from 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195) to ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193). New cstate: current_term: 5 leader_uuid: "ffe600335b884cae8bdd12c2ef74e670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:52.450250 16191 raft_consensus.cc:979] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826: : Instructing follower ffe600335b884cae8bdd12c2ef74e670 to start an election
I20260810 18:50:52.450819 16173 raft_consensus.cc:1067] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [term 2 LEADER]: Signalling peer ffe600335b884cae8bdd12c2ef74e670 to start an election
I20260810 18:50:52.452286 15612 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8e61e9bbff154604bce6712d94c73e90"
dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
from {username='slave'} at 127.0.0.1:60324
I20260810 18:50:52.452749 15612 raft_consensus.cc:482] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:52.453009 15612 raft_consensus.cc:3037] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:52.457333 15612 raft_consensus.cc:504] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:52.458564 15612 leader_election.cc:288] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 3 election: Requested vote from peers 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:52.459520 15753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e61e9bbff154604bce6712d94c73e90" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
I20260810 18:50:52.459815 15876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e61e9bbff154604bce6712d94c73e90" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
I20260810 18:50:52.460116 15753 raft_consensus.cc:3032] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:50:52.460374 15876 raft_consensus.cc:3037] T 8e61e9bbff154604bce6712d94c73e90 P 51eee958aa8948b1a46aa1641638b9ec [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:52.460449 15753 raft_consensus.cc:729] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2d28250046da4d688d76be4ae6e4c826, State: Running, Role: LEADER
I20260810 18:50:52.461484 15753 consensus_queue.cc:260] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:52.462491 15753 raft_consensus.cc:3037] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:52.465982 15876 raft_consensus.cc:2454] T 8e61e9bbff154604bce6712d94c73e90 P 51eee958aa8948b1a46aa1641638b9ec [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffe600335b884cae8bdd12c2ef74e670 in term 3.
I20260810 18:50:52.466900 15537 leader_election.cc:302] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51eee958aa8948b1a46aa1641638b9ec, ffe600335b884cae8bdd12c2ef74e670; no voters:
I20260810 18:50:52.466913 15753 raft_consensus.cc:2454] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffe600335b884cae8bdd12c2ef74e670 in term 3.
I20260810 18:50:52.467723 16175 raft_consensus.cc:2781] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:50:52.468144 16175 raft_consensus.cc:686] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [term 3 LEADER]: Becoming Leader. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Running, Role: LEADER
I20260810 18:50:52.468963 16175 consensus_queue.cc:237] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:52.476343 15415 catalog_manager.cc:5495] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 reported cstate change: term changed from 2 to 3, leader changed from 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194) to ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193). New cstate: current_term: 3 leader_uuid: "ffe600335b884cae8bdd12c2ef74e670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:52.566645 16191 raft_consensus.cc:979] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826: : Instructing follower ffe600335b884cae8bdd12c2ef74e670 to start an election
I20260810 18:50:52.567121 16173 raft_consensus.cc:1067] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826 [term 2 LEADER]: Signalling peer ffe600335b884cae8bdd12c2ef74e670 to start an election
I20260810 18:50:52.568555 15612 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9764b4e886a24c9db19a64075920a794"
dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
from {username='slave'} at 127.0.0.1:60324
I20260810 18:50:52.568965 15612 raft_consensus.cc:482] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:52.569196 15612 raft_consensus.cc:3037] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:52.573904 15612 raft_consensus.cc:504] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:52.575372 15612 leader_election.cc:288] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 3 election: Requested vote from peers 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:52.576331 15753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9764b4e886a24c9db19a64075920a794" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
I20260810 18:50:52.576635 15876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9764b4e886a24c9db19a64075920a794" candidate_uuid: "ffe600335b884cae8bdd12c2ef74e670" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
I20260810 18:50:52.576833 15753 raft_consensus.cc:3032] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826 [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:50:52.577164 15876 raft_consensus.cc:3037] T 9764b4e886a24c9db19a64075920a794 P 51eee958aa8948b1a46aa1641638b9ec [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:52.577245 15753 raft_consensus.cc:729] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2d28250046da4d688d76be4ae6e4c826, State: Running, Role: LEADER
I20260810 18:50:52.578120 15753 consensus_queue.cc:260] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:52.579216 15753 raft_consensus.cc:3037] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:52.582242 15876 raft_consensus.cc:2454] T 9764b4e886a24c9db19a64075920a794 P 51eee958aa8948b1a46aa1641638b9ec [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffe600335b884cae8bdd12c2ef74e670 in term 3.
I20260810 18:50:52.583287 15537 leader_election.cc:302] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51eee958aa8948b1a46aa1641638b9ec, ffe600335b884cae8bdd12c2ef74e670; no voters:
I20260810 18:50:52.584054 16175 raft_consensus.cc:2781] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:50:52.584477 16175 raft_consensus.cc:686] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [term 3 LEADER]: Becoming Leader. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Running, Role: LEADER
I20260810 18:50:52.584651 15753 raft_consensus.cc:2454] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffe600335b884cae8bdd12c2ef74e670 in term 3.
I20260810 18:50:52.585443 16175 consensus_queue.cc:237] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:52.592067 15415 catalog_manager.cc:5495] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 reported cstate change: term changed from 2 to 3, leader changed from 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194) to ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193). New cstate: current_term: 3 leader_uuid: "ffe600335b884cae8bdd12c2ef74e670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:52.718075 15876 raft_consensus.cc:1261] T 2813a19d92094a648be20c4c20f713fa P 51eee958aa8948b1a46aa1641638b9ec [term 2 FOLLOWER]: Refusing update from remote peer ffe600335b884cae8bdd12c2ef74e670: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:52.719261 16175 consensus_queue.cc:1035] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:50:52.729700 15753 raft_consensus.cc:1261] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [term 2 FOLLOWER]: Refusing update from remote peer ffe600335b884cae8bdd12c2ef74e670: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:52.731848 16203 consensus_queue.cc:1035] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:50:52.735658 15876 raft_consensus.cc:1261] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 4 FOLLOWER]: Refusing update from remote peer ffe600335b884cae8bdd12c2ef74e670: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260810 18:50:52.737180 16203 consensus_queue.cc:1035] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:50:52.750528 15753 raft_consensus.cc:1261] T 354460a175b24dcd873c370589a0f7d5 P 2d28250046da4d688d76be4ae6e4c826 [term 4 FOLLOWER]: Refusing update from remote peer ffe600335b884cae8bdd12c2ef74e670: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260810 18:50:52.751900 16203 consensus_queue.cc:1035] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:50:52.814421 15876 raft_consensus.cc:1261] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 5 FOLLOWER]: Refusing update from remote peer ffe600335b884cae8bdd12c2ef74e670: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260810 18:50:52.815681 16203 consensus_queue.cc:1035] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:50:52.824196 15753 raft_consensus.cc:1261] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [term 5 FOLLOWER]: Refusing update from remote peer ffe600335b884cae8bdd12c2ef74e670: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260810 18:50:52.825634 16175 consensus_queue.cc:1035] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:50:52.866365 15876 raft_consensus.cc:1261] T 8e61e9bbff154604bce6712d94c73e90 P 51eee958aa8948b1a46aa1641638b9ec [term 3 FOLLOWER]: Refusing update from remote peer ffe600335b884cae8bdd12c2ef74e670: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:52.867439 16203 consensus_queue.cc:1035] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:50:52.876299 15753 raft_consensus.cc:1261] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [term 3 FOLLOWER]: Refusing update from remote peer ffe600335b884cae8bdd12c2ef74e670: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:52.877977 16175 consensus_queue.cc:1035] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:50:52.984827 15753 raft_consensus.cc:1261] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826 [term 3 FOLLOWER]: Refusing update from remote peer ffe600335b884cae8bdd12c2ef74e670: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:52.986310 16175 consensus_queue.cc:1035] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:50:52.995751 15876 raft_consensus.cc:1261] T 9764b4e886a24c9db19a64075920a794 P 51eee958aa8948b1a46aa1641638b9ec [term 3 FOLLOWER]: Refusing update from remote peer ffe600335b884cae8bdd12c2ef74e670: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:52.997421 16217 consensus_queue.cc:1035] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:50:55.115691 14467 auto_leader_rebalancer-test.cc:373] The leader distribution is
I20260810 18:50:55.116569 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:55.124080 15612 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "20f91c7cf7b14e6ab5215e47458b4f1b"
dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
mode: GRACEFUL
new_leader_uuid: "2d28250046da4d688d76be4ae6e4c826"
from {username='slave'} at 127.0.0.1:60338
I20260810 18:50:55.124790 15612 raft_consensus.cc:595] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670 [term 2 LEADER]: Received request to transfer leadership to TS 2d28250046da4d688d76be4ae6e4c826
I20260810 18:50:55.127214 15612 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2813a19d92094a648be20c4c20f713fa"
dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
mode: GRACEFUL
new_leader_uuid: "2d28250046da4d688d76be4ae6e4c826"
from {username='slave'} at 127.0.0.1:60338
I20260810 18:50:55.127635 15612 raft_consensus.cc:595] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670 [term 2 LEADER]: Received request to transfer leadership to TS 2d28250046da4d688d76be4ae6e4c826
I20260810 18:50:55.129675 15612 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4f6b952757814de89b38a47f3942ee7e"
dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
mode: GRACEFUL
new_leader_uuid: "51eee958aa8948b1a46aa1641638b9ec"
from {username='slave'} at 127.0.0.1:60338
I20260810 18:50:55.130069 15612 raft_consensus.cc:595] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670 [term 2 LEADER]: Received request to transfer leadership to TS 51eee958aa8948b1a46aa1641638b9ec
I20260810 18:50:55.132110 15612 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8b96127e94b74666aa7a5d280228ba7d"
dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
mode: GRACEFUL
new_leader_uuid: "51eee958aa8948b1a46aa1641638b9ec"
from {username='slave'} at 127.0.0.1:60338
I20260810 18:50:55.132534 15612 raft_consensus.cc:595] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [term 2 LEADER]: Received request to transfer leadership to TS 51eee958aa8948b1a46aa1641638b9ec
I20260810 18:50:55.133415 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260810 18:50:55.133845 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:55.291118 16240 raft_consensus.cc:979] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670: : Instructing follower 51eee958aa8948b1a46aa1641638b9ec to start an election
I20260810 18:50:55.291747 16203 raft_consensus.cc:1067] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670 [term 2 LEADER]: Signalling peer 51eee958aa8948b1a46aa1641638b9ec to start an election
I20260810 18:50:55.293318 15879 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f6b952757814de89b38a47f3942ee7e"
dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
from {username='slave'} at 127.0.0.1:35178
I20260810 18:50:55.293773 15879 raft_consensus.cc:482] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:55.294106 15879 raft_consensus.cc:3037] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:55.299705 15879 raft_consensus.cc:504] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:55.302317 15879 leader_election.cc:288] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 3 election: Requested vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629)
I20260810 18:50:55.303462 15753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f6b952757814de89b38a47f3942ee7e" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
I20260810 18:50:55.304098 15753 raft_consensus.cc:3037] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:55.303290 15612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f6b952757814de89b38a47f3942ee7e" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
I20260810 18:50:55.304535 15612 raft_consensus.cc:3032] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670 [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:50:55.304842 15612 raft_consensus.cc:729] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Running, Role: LEADER
I20260810 18:50:55.305590 15612 consensus_queue.cc:260] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:55.306869 15612 raft_consensus.cc:3037] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:55.311347 15753 raft_consensus.cc:2454] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51eee958aa8948b1a46aa1641638b9ec in term 3.
I20260810 18:50:55.312615 15816 leader_election.cc:302] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d28250046da4d688d76be4ae6e4c826, 51eee958aa8948b1a46aa1641638b9ec; no voters:
I20260810 18:50:55.313493 16229 raft_consensus.cc:2781] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:50:55.313730 15612 raft_consensus.cc:2454] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51eee958aa8948b1a46aa1641638b9ec in term 3.
I20260810 18:50:55.314659 16229 raft_consensus.cc:686] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec [term 3 LEADER]: Becoming Leader. State: Replica: 51eee958aa8948b1a46aa1641638b9ec, State: Running, Role: LEADER
I20260810 18:50:55.315528 16229 consensus_queue.cc:237] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:55.322928 15417 catalog_manager.cc:5495] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec reported cstate change: term changed from 2 to 3, leader changed from ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193) to 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195). New cstate: current_term: 3 leader_uuid: "51eee958aa8948b1a46aa1641638b9ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } health_report { overall_health: HEALTHY } } }
I20260810 18:50:55.539135 16240 raft_consensus.cc:979] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670: : Instructing follower 51eee958aa8948b1a46aa1641638b9ec to start an election
I20260810 18:50:55.539641 16203 raft_consensus.cc:1067] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [term 2 LEADER]: Signalling peer 51eee958aa8948b1a46aa1641638b9ec to start an election
I20260810 18:50:55.541090 15879 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8b96127e94b74666aa7a5d280228ba7d"
dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
from {username='slave'} at 127.0.0.1:35178
I20260810 18:50:55.541512 15879 raft_consensus.cc:482] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:55.541746 15879 raft_consensus.cc:3037] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:55.546160 15879 raft_consensus.cc:504] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } }
I20260810 18:50:55.547410 15879 leader_election.cc:288] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 3 election: Requested vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629)
I20260810 18:50:55.548373 15612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b96127e94b74666aa7a5d280228ba7d" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
I20260810 18:50:55.548373 15753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b96127e94b74666aa7a5d280228ba7d" candidate_uuid: "51eee958aa8948b1a46aa1641638b9ec" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
I20260810 18:50:55.549105 15612 raft_consensus.cc:3032] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:50:55.549118 15753 raft_consensus.cc:3037] T 8b96127e94b74666aa7a5d280228ba7d P 2d28250046da4d688d76be4ae6e4c826 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:55.549548 15612 raft_consensus.cc:729] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Running, Role: LEADER
I20260810 18:50:55.550273 15612 consensus_queue.cc:260] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } }
I20260810 18:50:55.551451 15612 raft_consensus.cc:3037] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:55.555603 15753 raft_consensus.cc:2454] T 8b96127e94b74666aa7a5d280228ba7d P 2d28250046da4d688d76be4ae6e4c826 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51eee958aa8948b1a46aa1641638b9ec in term 3.
I20260810 18:50:55.556578 15816 leader_election.cc:302] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d28250046da4d688d76be4ae6e4c826, 51eee958aa8948b1a46aa1641638b9ec; no voters:
I20260810 18:50:55.557232 16229 raft_consensus.cc:2781] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:50:55.557483 15612 raft_consensus.cc:2454] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51eee958aa8948b1a46aa1641638b9ec in term 3.
I20260810 18:50:55.557724 16229 raft_consensus.cc:686] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [term 3 LEADER]: Becoming Leader. State: Replica: 51eee958aa8948b1a46aa1641638b9ec, State: Running, Role: LEADER
I20260810 18:50:55.558734 16229 consensus_queue.cc:237] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } }
I20260810 18:50:55.566263 15417 catalog_manager.cc:5495] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec reported cstate change: term changed from 2 to 3, leader changed from ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193) to 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195). New cstate: current_term: 3 leader_uuid: "51eee958aa8948b1a46aa1641638b9ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:55.608672 16240 raft_consensus.cc:979] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670: : Instructing follower 2d28250046da4d688d76be4ae6e4c826 to start an election
I20260810 18:50:55.609153 16203 raft_consensus.cc:1067] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670 [term 2 LEADER]: Signalling peer 2d28250046da4d688d76be4ae6e4c826 to start an election
I20260810 18:50:55.610416 15753 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2813a19d92094a648be20c4c20f713fa"
dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
from {username='slave'} at 127.0.0.1:59498
I20260810 18:50:55.610813 15753 raft_consensus.cc:482] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:55.611018 15753 raft_consensus.cc:3037] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:55.615017 15753 raft_consensus.cc:504] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:55.616724 15753 leader_election.cc:288] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 3 election: Requested vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:55.617555 15612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2813a19d92094a648be20c4c20f713fa" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
I20260810 18:50:55.617817 15879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2813a19d92094a648be20c4c20f713fa" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
I20260810 18:50:55.618016 15612 raft_consensus.cc:3032] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670 [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:50:55.618372 15612 raft_consensus.cc:729] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Running, Role: LEADER
I20260810 18:50:55.618378 15879 raft_consensus.cc:3037] T 2813a19d92094a648be20c4c20f713fa P 51eee958aa8948b1a46aa1641638b9ec [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:55.619112 15612 consensus_queue.cc:260] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:55.620043 15612 raft_consensus.cc:3037] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:55.623747 15879 raft_consensus.cc:2454] T 2813a19d92094a648be20c4c20f713fa P 51eee958aa8948b1a46aa1641638b9ec [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d28250046da4d688d76be4ae6e4c826 in term 3.
I20260810 18:50:55.624871 15673 leader_election.cc:302] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d28250046da4d688d76be4ae6e4c826, 51eee958aa8948b1a46aa1641638b9ec; no voters:
I20260810 18:50:55.625109 15612 raft_consensus.cc:2454] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d28250046da4d688d76be4ae6e4c826 in term 3.
I20260810 18:50:55.625844 16251 raft_consensus.cc:2781] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:50:55.626983 16251 raft_consensus.cc:686] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [term 3 LEADER]: Becoming Leader. State: Replica: 2d28250046da4d688d76be4ae6e4c826, State: Running, Role: LEADER
I20260810 18:50:55.627900 16251 consensus_queue.cc:237] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:55.634411 15417 catalog_manager.cc:5495] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 reported cstate change: term changed from 2 to 3, leader changed from ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193) to 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194). New cstate: current_term: 3 leader_uuid: "2d28250046da4d688d76be4ae6e4c826" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:55.696950 16240 raft_consensus.cc:979] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670: : Instructing follower 2d28250046da4d688d76be4ae6e4c826 to start an election
I20260810 18:50:55.697393 16203 raft_consensus.cc:1067] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670 [term 2 LEADER]: Signalling peer 2d28250046da4d688d76be4ae6e4c826 to start an election
I20260810 18:50:55.698724 15753 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "20f91c7cf7b14e6ab5215e47458b4f1b"
dest_uuid: "2d28250046da4d688d76be4ae6e4c826"
from {username='slave'} at 127.0.0.1:59498
I20260810 18:50:55.699237 15753 raft_consensus.cc:482] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:50:55.699519 15753 raft_consensus.cc:3037] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:55.703614 15753 raft_consensus.cc:504] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:55.704923 15753 leader_election.cc:288] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 3 election: Requested vote from peers ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539), 51eee958aa8948b1a46aa1641638b9ec (127.14.32.195:38891)
I20260810 18:50:55.705722 15612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f91c7cf7b14e6ab5215e47458b4f1b" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ffe600335b884cae8bdd12c2ef74e670"
I20260810 18:50:55.705895 15879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f91c7cf7b14e6ab5215e47458b4f1b" candidate_uuid: "2d28250046da4d688d76be4ae6e4c826" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "51eee958aa8948b1a46aa1641638b9ec"
I20260810 18:50:55.706269 15612 raft_consensus.cc:3032] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670 [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:50:55.706349 15879 raft_consensus.cc:3037] T 20f91c7cf7b14e6ab5215e47458b4f1b P 51eee958aa8948b1a46aa1641638b9ec [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:55.706609 15612 raft_consensus.cc:729] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ffe600335b884cae8bdd12c2ef74e670, State: Running, Role: LEADER
I20260810 18:50:55.707326 15612 consensus_queue.cc:260] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:55.708459 15612 raft_consensus.cc:3037] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:55.712276 15879 raft_consensus.cc:2454] T 20f91c7cf7b14e6ab5215e47458b4f1b P 51eee958aa8948b1a46aa1641638b9ec [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d28250046da4d688d76be4ae6e4c826 in term 3.
I20260810 18:50:55.713259 15673 leader_election.cc:302] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d28250046da4d688d76be4ae6e4c826, 51eee958aa8948b1a46aa1641638b9ec; no voters:
I20260810 18:50:55.713989 16251 raft_consensus.cc:2781] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:50:55.714039 15612 raft_consensus.cc:2454] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d28250046da4d688d76be4ae6e4c826 in term 3.
I20260810 18:50:55.714599 16251 raft_consensus.cc:686] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [term 3 LEADER]: Becoming Leader. State: Replica: 2d28250046da4d688d76be4ae6e4c826, State: Running, Role: LEADER
I20260810 18:50:55.715564 16251 consensus_queue.cc:237] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } }
I20260810 18:50:55.722343 15417 catalog_manager.cc:5495] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 reported cstate change: term changed from 2 to 3, leader changed from ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193) to 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194). New cstate: current_term: 3 leader_uuid: "2d28250046da4d688d76be4ae6e4c826" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:55.740504 15612 raft_consensus.cc:1261] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670 [term 3 FOLLOWER]: Refusing update from remote peer 51eee958aa8948b1a46aa1641638b9ec: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:55.741639 16229 consensus_queue.cc:1035] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:50:55.752449 15753 raft_consensus.cc:1261] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [term 3 FOLLOWER]: Refusing update from remote peer 51eee958aa8948b1a46aa1641638b9ec: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:55.753852 16229 consensus_queue.cc:1035] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:50:55.965706 15753 raft_consensus.cc:1261] T 8b96127e94b74666aa7a5d280228ba7d P 2d28250046da4d688d76be4ae6e4c826 [term 3 FOLLOWER]: Refusing update from remote peer 51eee958aa8948b1a46aa1641638b9ec: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:55.967020 16229 consensus_queue.cc:1035] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d28250046da4d688d76be4ae6e4c826" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:50:55.977401 15612 raft_consensus.cc:1261] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [term 3 FOLLOWER]: Refusing update from remote peer 51eee958aa8948b1a46aa1641638b9ec: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:55.978762 16270 consensus_queue.cc:1035] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:50:56.134758 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:56.141564 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:56.142012 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:56.152690 15879 raft_consensus.cc:1261] T 2813a19d92094a648be20c4c20f713fa P 51eee958aa8948b1a46aa1641638b9ec [term 3 FOLLOWER]: Refusing update from remote peer 2d28250046da4d688d76be4ae6e4c826: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:56.153957 16251 consensus_queue.cc:1035] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:50:56.163282 15612 raft_consensus.cc:1261] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670 [term 3 FOLLOWER]: Refusing update from remote peer 2d28250046da4d688d76be4ae6e4c826: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:56.165024 16251 consensus_queue.cc:1035] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:50:56.173863 15612 raft_consensus.cc:1261] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670 [term 3 FOLLOWER]: Refusing update from remote peer 2d28250046da4d688d76be4ae6e4c826: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:56.175094 16251 consensus_queue.cc:1035] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffe600335b884cae8bdd12c2ef74e670" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 36539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:50:56.184037 15879 raft_consensus.cc:1261] T 20f91c7cf7b14e6ab5215e47458b4f1b P 51eee958aa8948b1a46aa1641638b9ec [term 3 FOLLOWER]: Refusing update from remote peer 2d28250046da4d688d76be4ae6e4c826: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:56.185837 16280 consensus_queue.cc:1035] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51eee958aa8948b1a46aa1641638b9ec" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:50:57.142819 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:57.148484 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:57.148831 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:58.149644 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:58.155402 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:58.155738 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:50:59.156481 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:50:59.162365 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:50:59.162700 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:51:00.163551 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:51:00.169064 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:51:00.169437 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:51:01.170219 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:51:01.177103 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:51:01.177451 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:51:02.178344 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:51:02.185573 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:51:02.186066 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:51:03.187214 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:51:03.193042 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:51:03.193492 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:51:04.194566 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:51:04.202423 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260810 18:51:04.203024 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:51:05.232947 14467 auto_leader_rebalancer-test.cc:384] The leader distribution is
I20260810 18:51:05.233649 14467 tablet_server.cc:178] TabletServer@127.14.32.193:0 shutting down...
I20260810 18:51:05.265870 14467 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:51:05.266641 14467 tablet_replica.cc:330] T cd4103b9a0414848b8ad7e116bf4ded8 P ffe600335b884cae8bdd12c2ef74e670: stopping tablet replica
I20260810 18:51:05.267807 14467 raft_consensus.cc:2229] T cd4103b9a0414848b8ad7e116bf4ded8 P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:05.268574 14467 raft_consensus.cc:2258] T cd4103b9a0414848b8ad7e116bf4ded8 P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 18:51:05.269340 15675 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 3 similar messages]
I20260810 18:51:05.271994 14467 tablet_replica.cc:330] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670: stopping tablet replica
I20260810 18:51:05.273403 14467 raft_consensus.cc:2229] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [term 3 FOLLOWER]: Raft consensus shutting down.
W20260810 18:51:05.273993 15675 consensus_peers.cc:487] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 -> Peer ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539): Couldn't send request to peer ffe600335b884cae8bdd12c2ef74e670. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260810 18:51:05.275429 15817 consensus_peers.cc:487] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec -> Peer ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539): Couldn't send request to peer ffe600335b884cae8bdd12c2ef74e670. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:05.274045 14467 raft_consensus.cc:2258] T 8b96127e94b74666aa7a5d280228ba7d P ffe600335b884cae8bdd12c2ef74e670 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.278693 14467 tablet_replica.cc:330] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670: stopping tablet replica
I20260810 18:51:05.279382 14467 raft_consensus.cc:2229] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [term 4 LEADER]: Raft consensus shutting down.
I20260810 18:51:05.280324 14467 raft_consensus.cc:2258] T 354460a175b24dcd873c370589a0f7d5 P ffe600335b884cae8bdd12c2ef74e670 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.282215 14467 tablet_replica.cc:330] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670: stopping tablet replica
I20260810 18:51:05.282756 14467 raft_consensus.cc:2229] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 5 LEADER]: Raft consensus shutting down.
I20260810 18:51:05.283641 14467 raft_consensus.cc:2258] T e7e5673440be4239a048129ffad7a53c P ffe600335b884cae8bdd12c2ef74e670 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.285637 14467 tablet_replica.cc:330] T 6df3bb273c824e68b83d4f4afdcb5b66 P ffe600335b884cae8bdd12c2ef74e670: stopping tablet replica
I20260810 18:51:05.286128 14467 raft_consensus.cc:2229] T 6df3bb273c824e68b83d4f4afdcb5b66 P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:05.286628 14467 raft_consensus.cc:2258] T 6df3bb273c824e68b83d4f4afdcb5b66 P ffe600335b884cae8bdd12c2ef74e670 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.288322 14467 tablet_replica.cc:330] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670: stopping tablet replica
I20260810 18:51:05.288841 14467 raft_consensus.cc:2229] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:05.289250 14467 raft_consensus.cc:2258] T 4f6b952757814de89b38a47f3942ee7e P ffe600335b884cae8bdd12c2ef74e670 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.291234 14467 tablet_replica.cc:330] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670: stopping tablet replica
I20260810 18:51:05.291718 14467 raft_consensus.cc:2229] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:05.292128 14467 raft_consensus.cc:2258] T 20f91c7cf7b14e6ab5215e47458b4f1b P ffe600335b884cae8bdd12c2ef74e670 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.293888 14467 tablet_replica.cc:330] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670: stopping tablet replica
I20260810 18:51:05.294358 14467 raft_consensus.cc:2229] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [term 3 LEADER]: Raft consensus shutting down.
I20260810 18:51:05.295312 14467 raft_consensus.cc:2258] T 8e61e9bbff154604bce6712d94c73e90 P ffe600335b884cae8bdd12c2ef74e670 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.297019 14467 tablet_replica.cc:330] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670: stopping tablet replica
I20260810 18:51:05.297498 14467 raft_consensus.cc:2229] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:05.297935 14467 raft_consensus.cc:2258] T 2813a19d92094a648be20c4c20f713fa P ffe600335b884cae8bdd12c2ef74e670 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.299813 14467 tablet_replica.cc:330] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670: stopping tablet replica
I20260810 18:51:05.300310 14467 raft_consensus.cc:2229] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [term 3 LEADER]: Raft consensus shutting down.
I20260810 18:51:05.301113 14467 raft_consensus.cc:2258] T 9764b4e886a24c9db19a64075920a794 P ffe600335b884cae8bdd12c2ef74e670 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.356135 14467 tablet_server.cc:194] TabletServer@127.14.32.193:0 shutdown complete.
I20260810 18:51:05.376153 14467 tablet_server.cc:178] TabletServer@127.14.32.194:0 shutting down...
W20260810 18:51:05.389895 15816 consensus_peers.cc:487] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec -> Peer 2d28250046da4d688d76be4ae6e4c826 (127.14.32.194:46629): Couldn't send request to peer 2d28250046da4d688d76be4ae6e4c826. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:05.412328 14467 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:51:05.413215 14467 tablet_replica.cc:330] T cd4103b9a0414848b8ad7e116bf4ded8 P 2d28250046da4d688d76be4ae6e4c826: stopping tablet replica
I20260810 18:51:05.414171 14467 raft_consensus.cc:2229] T cd4103b9a0414848b8ad7e116bf4ded8 P 2d28250046da4d688d76be4ae6e4c826 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:51:05.415184 14467 raft_consensus.cc:2258] T cd4103b9a0414848b8ad7e116bf4ded8 P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.417050 14467 tablet_replica.cc:330] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826: stopping tablet replica
I20260810 18:51:05.417548 14467 raft_consensus.cc:2229] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:05.418023 14467 raft_consensus.cc:2258] T e7e5673440be4239a048129ffad7a53c P 2d28250046da4d688d76be4ae6e4c826 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.419869 14467 tablet_replica.cc:330] T 354460a175b24dcd873c370589a0f7d5 P 2d28250046da4d688d76be4ae6e4c826: stopping tablet replica
I20260810 18:51:05.420378 14467 raft_consensus.cc:2229] T 354460a175b24dcd873c370589a0f7d5 P 2d28250046da4d688d76be4ae6e4c826 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:05.420846 14467 raft_consensus.cc:2258] T 354460a175b24dcd873c370589a0f7d5 P 2d28250046da4d688d76be4ae6e4c826 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.422447 14467 tablet_replica.cc:330] T 6df3bb273c824e68b83d4f4afdcb5b66 P 2d28250046da4d688d76be4ae6e4c826: stopping tablet replica
I20260810 18:51:05.422919 14467 raft_consensus.cc:2229] T 6df3bb273c824e68b83d4f4afdcb5b66 P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:05.423355 14467 raft_consensus.cc:2258] T 6df3bb273c824e68b83d4f4afdcb5b66 P 2d28250046da4d688d76be4ae6e4c826 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.425042 14467 tablet_replica.cc:330] T 8b96127e94b74666aa7a5d280228ba7d P 2d28250046da4d688d76be4ae6e4c826: stopping tablet replica
I20260810 18:51:05.425508 14467 raft_consensus.cc:2229] T 8b96127e94b74666aa7a5d280228ba7d P 2d28250046da4d688d76be4ae6e4c826 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:05.425926 14467 raft_consensus.cc:2258] T 8b96127e94b74666aa7a5d280228ba7d P 2d28250046da4d688d76be4ae6e4c826 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.427675 14467 tablet_replica.cc:330] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826: stopping tablet replica
I20260810 18:51:05.428202 14467 raft_consensus.cc:2229] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:05.428630 14467 raft_consensus.cc:2258] T 9764b4e886a24c9db19a64075920a794 P 2d28250046da4d688d76be4ae6e4c826 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.430174 14467 tablet_replica.cc:330] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826: stopping tablet replica
I20260810 18:51:05.430641 14467 raft_consensus.cc:2229] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [term 3 LEADER]: Raft consensus shutting down.
I20260810 18:51:05.431535 14467 raft_consensus.cc:2258] T 2813a19d92094a648be20c4c20f713fa P 2d28250046da4d688d76be4ae6e4c826 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.433176 14467 tablet_replica.cc:330] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826: stopping tablet replica
I20260810 18:51:05.433768 14467 raft_consensus.cc:2229] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:05.434201 14467 raft_consensus.cc:2258] T 8e61e9bbff154604bce6712d94c73e90 P 2d28250046da4d688d76be4ae6e4c826 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.435793 14467 tablet_replica.cc:330] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826: stopping tablet replica
I20260810 18:51:05.436188 14467 raft_consensus.cc:2229] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [term 3 LEADER]: Raft consensus shutting down.
I20260810 18:51:05.436908 14467 raft_consensus.cc:2258] T 20f91c7cf7b14e6ab5215e47458b4f1b P 2d28250046da4d688d76be4ae6e4c826 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.438478 14467 tablet_replica.cc:330] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826: stopping tablet replica
I20260810 18:51:05.438891 14467 raft_consensus.cc:2229] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:05.439337 14467 raft_consensus.cc:2258] T 4f6b952757814de89b38a47f3942ee7e P 2d28250046da4d688d76be4ae6e4c826 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.468046 14467 tablet_server.cc:194] TabletServer@127.14.32.194:0 shutdown complete.
W20260810 18:51:05.474468 15817 consensus_peers.cc:487] T 6df3bb273c824e68b83d4f4afdcb5b66 P 51eee958aa8948b1a46aa1641638b9ec -> Peer ffe600335b884cae8bdd12c2ef74e670 (127.14.32.193:36539): Couldn't send request to peer ffe600335b884cae8bdd12c2ef74e670. Status: Network error: Client connection negotiation failed: client connection to 127.14.32.193:36539: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:05.489821 14467 tablet_server.cc:178] TabletServer@127.14.32.195:0 shutting down...
I20260810 18:51:05.518302 14467 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:51:05.518985 14467 tablet_replica.cc:330] T 6df3bb273c824e68b83d4f4afdcb5b66 P 51eee958aa8948b1a46aa1641638b9ec: stopping tablet replica
I20260810 18:51:05.519676 14467 raft_consensus.cc:2229] T 6df3bb273c824e68b83d4f4afdcb5b66 P 51eee958aa8948b1a46aa1641638b9ec [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:51:05.520596 14467 raft_consensus.cc:2258] T 6df3bb273c824e68b83d4f4afdcb5b66 P 51eee958aa8948b1a46aa1641638b9ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.522575 14467 tablet_replica.cc:330] T 2813a19d92094a648be20c4c20f713fa P 51eee958aa8948b1a46aa1641638b9ec: stopping tablet replica
I20260810 18:51:05.523151 14467 raft_consensus.cc:2229] T 2813a19d92094a648be20c4c20f713fa P 51eee958aa8948b1a46aa1641638b9ec [term 3 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:05.523610 14467 raft_consensus.cc:2258] T 2813a19d92094a648be20c4c20f713fa P 51eee958aa8948b1a46aa1641638b9ec [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.525246 14467 tablet_replica.cc:330] T 9764b4e886a24c9db19a64075920a794 P 51eee958aa8948b1a46aa1641638b9ec: stopping tablet replica
I20260810 18:51:05.525758 14467 raft_consensus.cc:2229] T 9764b4e886a24c9db19a64075920a794 P 51eee958aa8948b1a46aa1641638b9ec [term 3 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:05.526165 14467 raft_consensus.cc:2258] T 9764b4e886a24c9db19a64075920a794 P 51eee958aa8948b1a46aa1641638b9ec [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.527851 14467 tablet_replica.cc:330] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec: stopping tablet replica
I20260810 18:51:05.528337 14467 raft_consensus.cc:2229] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec [term 3 LEADER]: Raft consensus shutting down.
I20260810 18:51:05.529102 14467 raft_consensus.cc:2258] T 4f6b952757814de89b38a47f3942ee7e P 51eee958aa8948b1a46aa1641638b9ec [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.530599 14467 tablet_replica.cc:330] T 8e61e9bbff154604bce6712d94c73e90 P 51eee958aa8948b1a46aa1641638b9ec: stopping tablet replica
I20260810 18:51:05.531105 14467 raft_consensus.cc:2229] T 8e61e9bbff154604bce6712d94c73e90 P 51eee958aa8948b1a46aa1641638b9ec [term 3 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:05.531582 14467 raft_consensus.cc:2258] T 8e61e9bbff154604bce6712d94c73e90 P 51eee958aa8948b1a46aa1641638b9ec [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.533159 14467 tablet_replica.cc:330] T 20f91c7cf7b14e6ab5215e47458b4f1b P 51eee958aa8948b1a46aa1641638b9ec: stopping tablet replica
I20260810 18:51:05.533628 14467 raft_consensus.cc:2229] T 20f91c7cf7b14e6ab5215e47458b4f1b P 51eee958aa8948b1a46aa1641638b9ec [term 3 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:05.534049 14467 raft_consensus.cc:2258] T 20f91c7cf7b14e6ab5215e47458b4f1b P 51eee958aa8948b1a46aa1641638b9ec [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.535604 14467 tablet_replica.cc:330] T cd4103b9a0414848b8ad7e116bf4ded8 P 51eee958aa8948b1a46aa1641638b9ec: stopping tablet replica
I20260810 18:51:05.536047 14467 raft_consensus.cc:2229] T cd4103b9a0414848b8ad7e116bf4ded8 P 51eee958aa8948b1a46aa1641638b9ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:05.536435 14467 raft_consensus.cc:2258] T cd4103b9a0414848b8ad7e116bf4ded8 P 51eee958aa8948b1a46aa1641638b9ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.537936 14467 tablet_replica.cc:330] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec: stopping tablet replica
I20260810 18:51:05.538424 14467 raft_consensus.cc:2229] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [term 3 LEADER]: Raft consensus shutting down.
I20260810 18:51:05.539333 14467 raft_consensus.cc:2258] T 8b96127e94b74666aa7a5d280228ba7d P 51eee958aa8948b1a46aa1641638b9ec [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.540913 14467 tablet_replica.cc:330] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec: stopping tablet replica
I20260810 18:51:05.541373 14467 raft_consensus.cc:2229] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 5 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:05.541798 14467 raft_consensus.cc:2258] T e7e5673440be4239a048129ffad7a53c P 51eee958aa8948b1a46aa1641638b9ec [term 5 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.543344 14467 tablet_replica.cc:330] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec: stopping tablet replica
I20260810 18:51:05.543781 14467 raft_consensus.cc:2229] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 4 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:05.544198 14467 raft_consensus.cc:2258] T 354460a175b24dcd873c370589a0f7d5 P 51eee958aa8948b1a46aa1641638b9ec [term 4 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.572939 14467 tablet_server.cc:194] TabletServer@127.14.32.195:0 shutdown complete.
I20260810 18:51:05.592969 14467 master.cc:537] Master@127.14.32.254:34643 shutting down...
I20260810 18:51:05.615147 14467 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 78d70a72464749988488cc4d84aeb847 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:51:05.615770 14467 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 78d70a72464749988488cc4d84aeb847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:05.616120 14467 tablet_replica.cc:330] T 00000000000000000000000000000000 P 78d70a72464749988488cc4d84aeb847: stopping tablet replica
I20260810 18:51:05.635838 14467 master.cc:552] Master@127.14.32.254:34643 shutdown complete.
2d28250046da4d688d76be4ae6e4c826 4
51eee958aa8948b1a46aa1641638b9ec 1
ffe600335b884cae8bdd12c2ef74e670 5
2d28250046da4d688d76be4ae6e4c826 4
51eee958aa8948b1a46aa1641638b9ec 3
ffe600335b884cae8bdd12c2ef74e670 3
2d28250046da4d688d76be4ae6e4c826 1
51eee958aa8948b1a46aa1641638b9ec 1
ffe600335b884cae8bdd12c2ef74e670 8
2d28250046da4d688d76be4ae6e4c826 3
51eee958aa8948b1a46aa1641638b9ec 3
ffe600335b884cae8bdd12c2ef74e670 4
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28720 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20260810 18:51:05.679561 14467 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.14.32.254:46351
I20260810 18:51:05.680640 14467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:51:05.685920 16304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:51:05.686442 16305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:51:05.687594 16307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:51:05.688585 14467 server_base.cc:1034] running on GCE node
I20260810 18:51:05.689646 14467 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:51:05.689829 14467 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:51:05.689939 14467 hybrid_clock.cc:648] HybridClock initialized: now 1786387865689930 us; error 0 us; skew 500 ppm
I20260810 18:51:05.690385 14467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:51:05.693141 14467 webserver.cc:460] Webserver started at http://127.14.32.254:33541/ using document root <none> and password file <none>
I20260810 18:51:05.693557 14467 fs_manager.cc:359] Metadata directory not provided
I20260810 18:51:05.693703 14467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:51:05.693928 14467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:51:05.695153 14467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/master-0-root/instance:
uuid: "0975c6fa18274f2baee7fb3615d5b985"
format_stamp: "Formatted at 2026-08-10 18:51:05 on dist-test-slave-n326"
I20260810 18:51:05.699481 14467 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260810 18:51:05.702608 16318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:51:05.703377 14467 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:51:05.703657 14467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/master-0-root
uuid: "0975c6fa18274f2baee7fb3615d5b985"
format_stamp: "Formatted at 2026-08-10 18:51:05 on dist-test-slave-n326"
I20260810 18:51:05.703920 14467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:51:05.729628 14467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:51:05.730937 14467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:51:05.770653 14467 rpc_server.cc:307] RPC server started. Bound to: 127.14.32.254:46351
I20260810 18:51:05.770761 16391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.32.254:46351 every 8 connection(s)
I20260810 18:51:05.774945 16393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:05.785517 16393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0975c6fa18274f2baee7fb3615d5b985: Bootstrap starting.
I20260810 18:51:05.790213 16393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0975c6fa18274f2baee7fb3615d5b985: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:05.794719 16393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0975c6fa18274f2baee7fb3615d5b985: No bootstrap required, opened a new log
I20260810 18:51:05.796902 16393 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0975c6fa18274f2baee7fb3615d5b985 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0975c6fa18274f2baee7fb3615d5b985" member_type: VOTER }
I20260810 18:51:05.797413 16393 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0975c6fa18274f2baee7fb3615d5b985 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:05.797605 16393 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0975c6fa18274f2baee7fb3615d5b985 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0975c6fa18274f2baee7fb3615d5b985, State: Initialized, Role: FOLLOWER
I20260810 18:51:05.798230 16393 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0975c6fa18274f2baee7fb3615d5b985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0975c6fa18274f2baee7fb3615d5b985" member_type: VOTER }
I20260810 18:51:05.798715 16393 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0975c6fa18274f2baee7fb3615d5b985 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:51:05.798923 16393 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0975c6fa18274f2baee7fb3615d5b985 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:51:05.799187 16393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0975c6fa18274f2baee7fb3615d5b985 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:05.803399 16393 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0975c6fa18274f2baee7fb3615d5b985 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0975c6fa18274f2baee7fb3615d5b985" member_type: VOTER }
I20260810 18:51:05.803879 16393 leader_election.cc:302] T 00000000000000000000000000000000 P 0975c6fa18274f2baee7fb3615d5b985 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0975c6fa18274f2baee7fb3615d5b985; no voters:
I20260810 18:51:05.805135 16393 leader_election.cc:288] T 00000000000000000000000000000000 P 0975c6fa18274f2baee7fb3615d5b985 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:51:05.805588 16397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0975c6fa18274f2baee7fb3615d5b985 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:05.807149 16397 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0975c6fa18274f2baee7fb3615d5b985 [term 1 LEADER]: Becoming Leader. State: Replica: 0975c6fa18274f2baee7fb3615d5b985, State: Running, Role: LEADER
I20260810 18:51:05.807844 16397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0975c6fa18274f2baee7fb3615d5b985 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0975c6fa18274f2baee7fb3615d5b985" member_type: VOTER }
I20260810 18:51:05.808621 16393 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0975c6fa18274f2baee7fb3615d5b985 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:51:05.813498 16402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0975c6fa18274f2baee7fb3615d5b985 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0975c6fa18274f2baee7fb3615d5b985. Latest consensus state: current_term: 1 leader_uuid: "0975c6fa18274f2baee7fb3615d5b985" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0975c6fa18274f2baee7fb3615d5b985" member_type: VOTER } }
I20260810 18:51:05.813717 16398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0975c6fa18274f2baee7fb3615d5b985 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0975c6fa18274f2baee7fb3615d5b985" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0975c6fa18274f2baee7fb3615d5b985" member_type: VOTER } }
I20260810 18:51:05.814077 16402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0975c6fa18274f2baee7fb3615d5b985 [sys.catalog]: This master's current role is: LEADER
I20260810 18:51:05.814344 16398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0975c6fa18274f2baee7fb3615d5b985 [sys.catalog]: This master's current role is: LEADER
I20260810 18:51:05.816272 16413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:51:05.823335 16413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:51:05.824450 14467 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260810 18:51:05.832480 16413 catalog_manager.cc:1347] Generated new cluster ID: 4ac46a793a4647e988ee59bdf3cbf496
I20260810 18:51:05.832803 16413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:51:05.853976 16413 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:51:05.855917 16413 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:51:05.870957 16413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0975c6fa18274f2baee7fb3615d5b985: Generated new TSK 0
I20260810 18:51:05.871785 16413 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:51:05.890934 14467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:51:05.897233 16428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:51:05.898761 16431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:51:05.900650 16435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:51:05.901460 14467 server_base.cc:1034] running on GCE node
I20260810 18:51:05.902514 14467 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:51:05.902720 14467 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:51:05.902894 14467 hybrid_clock.cc:648] HybridClock initialized: now 1786387865902876 us; error 0 us; skew 500 ppm
I20260810 18:51:05.903431 14467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:51:05.906212 14467 webserver.cc:460] Webserver started at http://127.14.32.193:33259/ using document root <none> and password file <none>
I20260810 18:51:05.906733 14467 fs_manager.cc:359] Metadata directory not provided
I20260810 18:51:05.906955 14467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:51:05.907308 14467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:51:05.908620 14467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-0-root/instance:
uuid: "d304ac47fdf846de81337057a4ef7bea"
format_stamp: "Formatted at 2026-08-10 18:51:05 on dist-test-slave-n326"
I20260810 18:51:05.913650 14467 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260810 18:51:05.917071 16442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:51:05.917963 14467 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:51:05.918272 14467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-0-root
uuid: "d304ac47fdf846de81337057a4ef7bea"
format_stamp: "Formatted at 2026-08-10 18:51:05 on dist-test-slave-n326"
I20260810 18:51:05.918532 14467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:51:05.940877 14467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:51:05.942138 14467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:51:05.943574 14467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:51:05.945878 14467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:51:05.946086 14467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:51:05.946338 14467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:51:05.946513 14467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:51:05.988062 14467 rpc_server.cc:307] RPC server started. Bound to: 127.14.32.193:40791
I20260810 18:51:05.988183 16545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.32.193:40791 every 8 connection(s)
I20260810 18:51:05.992975 14467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:51:06.000113 16551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:51:06.001566 16554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:51:06.005107 16556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:51:06.006335 14467 server_base.cc:1034] running on GCE node
I20260810 18:51:06.007831 14467 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:51:06.008119 14467 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:51:06.008334 14467 hybrid_clock.cc:648] HybridClock initialized: now 1786387866008316 us; error 0 us; skew 500 ppm
I20260810 18:51:06.009160 14467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:51:06.010067 16546 heartbeater.cc:347] Connected to a master server at 127.14.32.254:46351
I20260810 18:51:06.010556 16546 heartbeater.cc:464] Registering TS with master...
I20260810 18:51:06.011546 16546 heartbeater.cc:511] Master 127.14.32.254:46351 requested a full tablet report, sending...
I20260810 18:51:06.012534 14467 webserver.cc:460] Webserver started at http://127.14.32.194:46407/ using document root <none> and password file <none>
I20260810 18:51:06.013378 14467 fs_manager.cc:359] Metadata directory not provided
I20260810 18:51:06.013758 14467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:51:06.014077 14467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:51:06.014400 16345 ts_manager.cc:194] Registered new tserver with Master: d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791)
I20260810 18:51:06.015476 14467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-1-root/instance:
uuid: "0397d760589d40d3b92ef141cdfc2dbb"
format_stamp: "Formatted at 2026-08-10 18:51:06 on dist-test-slave-n326"
I20260810 18:51:06.016425 16345 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37022
I20260810 18:51:06.021620 14467 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 18:51:06.025101 16567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:51:06.026074 14467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:51:06.026326 14467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-1-root
uuid: "0397d760589d40d3b92ef141cdfc2dbb"
format_stamp: "Formatted at 2026-08-10 18:51:06 on dist-test-slave-n326"
I20260810 18:51:06.026592 14467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:51:06.049410 14467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:51:06.050714 14467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:51:06.052345 14467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:51:06.054610 14467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:51:06.054823 14467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:51:06.055111 14467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:51:06.055267 14467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:51:06.094390 14467 rpc_server.cc:307] RPC server started. Bound to: 127.14.32.194:45827
I20260810 18:51:06.094513 16671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.32.194:45827 every 8 connection(s)
I20260810 18:51:06.099135 14467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:51:06.106952 16678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:51:06.109668 16680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:51:06.111683 14467 server_base.cc:1034] running on GCE node
W20260810 18:51:06.112258 16683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:51:06.113409 14467 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:51:06.113687 14467 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:51:06.113884 14467 hybrid_clock.cc:648] HybridClock initialized: now 1786387866113869 us; error 0 us; skew 500 ppm
I20260810 18:51:06.114548 14467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:51:06.114750 16672 heartbeater.cc:347] Connected to a master server at 127.14.32.254:46351
I20260810 18:51:06.115163 16672 heartbeater.cc:464] Registering TS with master...
I20260810 18:51:06.115999 16672 heartbeater.cc:511] Master 127.14.32.254:46351 requested a full tablet report, sending...
I20260810 18:51:06.117364 14467 webserver.cc:460] Webserver started at http://127.14.32.195:34987/ using document root <none> and password file <none>
I20260810 18:51:06.118006 14467 fs_manager.cc:359] Metadata directory not provided
I20260810 18:51:06.118199 14467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:51:06.118170 16345 ts_manager.cc:194] Registered new tserver with Master: 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:06.118615 14467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:51:06.119845 14467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-2-root/instance:
uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
format_stamp: "Formatted at 2026-08-10 18:51:06 on dist-test-slave-n326"
I20260810 18:51:06.120029 16345 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37026
I20260810 18:51:06.124348 14467 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:51:06.127789 16691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:51:06.128667 14467 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260810 18:51:06.128979 14467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-2-root
uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
format_stamp: "Formatted at 2026-08-10 18:51:06 on dist-test-slave-n326"
I20260810 18:51:06.129287 14467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:51:06.142505 14467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:51:06.143792 14467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:51:06.145244 14467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:51:06.147523 14467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:51:06.147744 14467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:51:06.147974 14467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:51:06.148138 14467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:51:06.188048 14467 rpc_server.cc:307] RPC server started. Bound to: 127.14.32.195:40171
I20260810 18:51:06.188174 16786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.32.195:40171 every 8 connection(s)
I20260810 18:51:06.204929 16787 heartbeater.cc:347] Connected to a master server at 127.14.32.254:46351
I20260810 18:51:06.205474 16787 heartbeater.cc:464] Registering TS with master...
I20260810 18:51:06.206394 16787 heartbeater.cc:511] Master 127.14.32.254:46351 requested a full tablet report, sending...
I20260810 18:51:06.208421 16345 ts_manager.cc:194] Registered new tserver with Master: 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:06.208556 14467 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017352742s
I20260810 18:51:06.210736 16345 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37030
I20260810 18:51:06.211102 14467 test_util.cc:268] Using random seed: -701345047
I20260810 18:51:06.237090 16345 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37044:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:51:06.243139 16345 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:51:06.457536 16739 tablet_service.cc:1467] Processing CreateTablet for tablet b650a523bfc649d4a7ec8bfd04a67c43 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260810 18:51:06.458267 16735 tablet_service.cc:1467] Processing CreateTablet for tablet ccee0891d8df4f81beb2cfbe1b5485f2 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260810 18:51:06.460214 16736 tablet_service.cc:1467] Processing CreateTablet for tablet 42bb15fb54124b8db9b482e3f17fc87c (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260810 18:51:06.461398 16728 tablet_service.cc:1467] Processing CreateTablet for tablet 32567b0896594c1186a56d27e36ca0a0 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260810 18:51:06.462277 16733 tablet_service.cc:1467] Processing CreateTablet for tablet b1af6c49790c42fc9b6455aec5f0d6af (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260810 18:51:06.462698 16619 tablet_service.cc:1467] Processing CreateTablet for tablet b650a523bfc649d4a7ec8bfd04a67c43 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260810 18:51:06.463423 16726 tablet_service.cc:1467] Processing CreateTablet for tablet 0a481385212f4e189ea5b2f5e215be4d (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260810 18:51:06.464149 16619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b650a523bfc649d4a7ec8bfd04a67c43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.464306 16732 tablet_service.cc:1467] Processing CreateTablet for tablet dac7251004484853a9326a55ffbb9c6d (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260810 18:51:06.465314 16605 tablet_service.cc:1467] Processing CreateTablet for tablet 50fbe8baa0b345b0b2c4e4489e397660 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260810 18:51:06.466638 16605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50fbe8baa0b345b0b2c4e4489e397660. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.466603 16731 tablet_service.cc:1467] Processing CreateTablet for tablet 50fbe8baa0b345b0b2c4e4489e397660 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260810 18:51:06.468238 16614 tablet_service.cc:1467] Processing CreateTablet for tablet 42bb15fb54124b8db9b482e3f17fc87c (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260810 18:51:06.468922 16730 tablet_service.cc:1467] Processing CreateTablet for tablet b4082c2e6a774acfa97ec268831748a5 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260810 18:51:06.469559 16614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42bb15fb54124b8db9b482e3f17fc87c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.469388 16610 tablet_service.cc:1467] Processing CreateTablet for tablet b1af6c49790c42fc9b6455aec5f0d6af (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260810 18:51:06.471628 16611 tablet_service.cc:1467] Processing CreateTablet for tablet ccee0891d8df4f81beb2cfbe1b5485f2 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260810 18:51:06.472211 16604 tablet_service.cc:1467] Processing CreateTablet for tablet b4082c2e6a774acfa97ec268831748a5 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260810 18:51:06.472937 16611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccee0891d8df4f81beb2cfbe1b5485f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.473510 16604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4082c2e6a774acfa97ec268831748a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.468261 16609 tablet_service.cc:1467] Processing CreateTablet for tablet dac7251004484853a9326a55ffbb9c6d (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260810 18:51:06.475117 16609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dac7251004484853a9326a55ffbb9c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.475507 16494 tablet_service.cc:1467] Processing CreateTablet for tablet b650a523bfc649d4a7ec8bfd04a67c43 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260810 18:51:06.476208 16483 tablet_service.cc:1467] Processing CreateTablet for tablet dac7251004484853a9326a55ffbb9c6d (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260810 18:51:06.476771 16494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b650a523bfc649d4a7ec8bfd04a67c43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.477458 16483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dac7251004484853a9326a55ffbb9c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.478885 16493 tablet_service.cc:1467] Processing CreateTablet for tablet 42bb15fb54124b8db9b482e3f17fc87c (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260810 18:51:06.480443 16493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42bb15fb54124b8db9b482e3f17fc87c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.480346 16482 tablet_service.cc:1467] Processing CreateTablet for tablet 50fbe8baa0b345b0b2c4e4489e397660 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260810 18:51:06.482815 16478 tablet_service.cc:1467] Processing CreateTablet for tablet b4082c2e6a774acfa97ec268831748a5 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260810 18:51:06.484455 16610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1af6c49790c42fc9b6455aec5f0d6af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.484272 16492 tablet_service.cc:1467] Processing CreateTablet for tablet ccee0891d8df4f81beb2cfbe1b5485f2 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260810 18:51:06.485075 16477 tablet_service.cc:1467] Processing CreateTablet for tablet 9e960488563344af83b979c260c2dd06 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260810 18:51:06.486486 16486 tablet_service.cc:1467] Processing CreateTablet for tablet b1af6c49790c42fc9b6455aec5f0d6af (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260810 18:51:06.487247 16476 tablet_service.cc:1467] Processing CreateTablet for tablet 32567b0896594c1186a56d27e36ca0a0 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260810 18:51:06.489426 16475 tablet_service.cc:1467] Processing CreateTablet for tablet 0a481385212f4e189ea5b2f5e215be4d (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260810 18:51:06.490840 16482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50fbe8baa0b345b0b2c4e4489e397660. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.460215 16729 tablet_service.cc:1467] Processing CreateTablet for tablet 9e960488563344af83b979c260c2dd06 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260810 18:51:06.494004 16492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccee0891d8df4f81beb2cfbe1b5485f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.495448 16475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a481385212f4e189ea5b2f5e215be4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.499459 16486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1af6c49790c42fc9b6455aec5f0d6af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.459044 16739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b650a523bfc649d4a7ec8bfd04a67c43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.501564 16732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dac7251004484853a9326a55ffbb9c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.502252 16735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccee0891d8df4f81beb2cfbe1b5485f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.500970 16601 tablet_service.cc:1467] Processing CreateTablet for tablet 9e960488563344af83b979c260c2dd06 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260810 18:51:06.502907 16477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e960488563344af83b979c260c2dd06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.503387 16601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e960488563344af83b979c260c2dd06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.503473 16619 tablet_service.cc:1467] Processing CreateTablet for tablet 32567b0896594c1186a56d27e36ca0a0 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260810 18:51:06.504004 16476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32567b0896594c1186a56d27e36ca0a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.504982 16619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32567b0896594c1186a56d27e36ca0a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.505204 16478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4082c2e6a774acfa97ec268831748a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.503357 16726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a481385212f4e189ea5b2f5e215be4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.508628 16733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1af6c49790c42fc9b6455aec5f0d6af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.509907 16731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50fbe8baa0b345b0b2c4e4489e397660. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.511039 16730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4082c2e6a774acfa97ec268831748a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.512485 16729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e960488563344af83b979c260c2dd06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.513918 16736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42bb15fb54124b8db9b482e3f17fc87c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.516682 16728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32567b0896594c1186a56d27e36ca0a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.542707 16600 tablet_service.cc:1467] Processing CreateTablet for tablet 0a481385212f4e189ea5b2f5e215be4d (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260810 18:51:06.544129 16600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a481385212f4e189ea5b2f5e215be4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.554716 16619 tablet_service.cc:1467] Processing CreateTablet for tablet cbdf372e5e3b4efb80f86146885fe25e (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260810 18:51:06.556144 16619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbdf372e5e3b4efb80f86146885fe25e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.588546 16814 tablet_bootstrap.cc:492] T 42bb15fb54124b8db9b482e3f17fc87c P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:06.589675 16599 tablet_service.cc:1467] Processing CreateTablet for tablet 03cb267b222b440981c94d431e7273bf (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260810 18:51:06.606503 16599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03cb267b222b440981c94d431e7273bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.607860 16611 tablet_service.cc:1467] Processing CreateTablet for tablet 6fa467f263e640b095b31b6d998258db (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260810 18:51:06.609143 16611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fa467f263e640b095b31b6d998258db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.620146 16619 tablet_service.cc:1467] Processing CreateTablet for tablet 8547cf266dd24ff289d7d8ff5e2174d0 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260810 18:51:06.631342 16614 tablet_service.cc:1467] Processing CreateTablet for tablet 4c87160448924498ac7de68947b3dc19 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260810 18:51:06.632726 16614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c87160448924498ac7de68947b3dc19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.639307 16733 tablet_service.cc:1467] Processing CreateTablet for tablet cbdf372e5e3b4efb80f86146885fe25e (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260810 18:51:06.645246 16733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbdf372e5e3b4efb80f86146885fe25e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.652241 16818 tablet_bootstrap.cc:492] T 32567b0896594c1186a56d27e36ca0a0 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:06.656167 16605 tablet_service.cc:1467] Processing CreateTablet for tablet d16638c4ef6148f8a1e6ff4e81ae955d (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260810 18:51:06.655943 16735 tablet_service.cc:1467] Processing CreateTablet for tablet 6fa467f263e640b095b31b6d998258db (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260810 18:51:06.655875 16728 tablet_service.cc:1467] Processing CreateTablet for tablet 03cb267b222b440981c94d431e7273bf (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260810 18:51:06.649205 16619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8547cf266dd24ff289d7d8ff5e2174d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.661826 16731 tablet_service.cc:1467] Processing CreateTablet for tablet 8547cf266dd24ff289d7d8ff5e2174d0 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260810 18:51:06.663268 16735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fa467f263e640b095b31b6d998258db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.664301 16731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8547cf266dd24ff289d7d8ff5e2174d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.665627 16728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03cb267b222b440981c94d431e7273bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.669538 16609 tablet_service.cc:1467] Processing CreateTablet for tablet 9dc14894db5b4872b0a5dd8d933426f4 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260810 18:51:06.670370 16605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d16638c4ef6148f8a1e6ff4e81ae955d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.666549 16739 tablet_service.cc:1467] Processing CreateTablet for tablet 4c87160448924498ac7de68947b3dc19 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260810 18:51:06.702639 16726 tablet_service.cc:1467] Processing CreateTablet for tablet 9dc14894db5b4872b0a5dd8d933426f4 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260810 18:51:06.684481 16739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c87160448924498ac7de68947b3dc19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.680912 16494 tablet_service.cc:1467] Processing CreateTablet for tablet cbdf372e5e3b4efb80f86146885fe25e (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260810 18:51:06.705708 16609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dc14894db5b4872b0a5dd8d933426f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.688589 16814 tablet_bootstrap.cc:654] T 42bb15fb54124b8db9b482e3f17fc87c P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:06.712462 16494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbdf372e5e3b4efb80f86146885fe25e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.713603 16476 tablet_service.cc:1467] Processing CreateTablet for tablet 03cb267b222b440981c94d431e7273bf (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260810 18:51:06.714867 16476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03cb267b222b440981c94d431e7273bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.696636 16818 tablet_bootstrap.cc:654] T 32567b0896594c1186a56d27e36ca0a0 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:06.722043 16726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dc14894db5b4872b0a5dd8d933426f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.694666 16475 tablet_service.cc:1467] Processing CreateTablet for tablet 6fa467f263e640b095b31b6d998258db (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260810 18:51:06.697710 16610 tablet_service.cc:1467] Processing CreateTablet for tablet 91b6af1458cd44d8ba26b43579613ecb (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260810 18:51:06.746160 16817 tablet_bootstrap.cc:492] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:06.747151 16610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91b6af1458cd44d8ba26b43579613ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.749609 16730 tablet_service.cc:1467] Processing CreateTablet for tablet 91b6af1458cd44d8ba26b43579613ecb (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260810 18:51:06.700165 16736 tablet_service.cc:1467] Processing CreateTablet for tablet d16638c4ef6148f8a1e6ff4e81ae955d (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260810 18:51:06.750842 16730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91b6af1458cd44d8ba26b43579613ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.753684 16492 tablet_service.cc:1467] Processing CreateTablet for tablet 9dc14894db5b4872b0a5dd8d933426f4 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260810 18:51:06.746686 16611 tablet_service.cc:1467] Processing CreateTablet for tablet ef3f6c02785c4a76ba768b20f41f2dcd (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260810 18:51:06.754606 16736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d16638c4ef6148f8a1e6ff4e81ae955d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.755988 16611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef3f6c02785c4a76ba768b20f41f2dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.756332 16604 tablet_service.cc:1467] Processing CreateTablet for tablet 3e58f8d2231b4e2d900ec099ae700d09 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260810 18:51:06.757517 16604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e58f8d2231b4e2d900ec099ae700d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.742214 16486 tablet_service.cc:1467] Processing CreateTablet for tablet 4c87160448924498ac7de68947b3dc19 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260810 18:51:06.754837 16729 tablet_service.cc:1467] Processing CreateTablet for tablet ef3f6c02785c4a76ba768b20f41f2dcd (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260810 18:51:06.775300 16492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dc14894db5b4872b0a5dd8d933426f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.775820 16729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef3f6c02785c4a76ba768b20f41f2dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.777987 16475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fa467f263e640b095b31b6d998258db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.764292 16732 tablet_service.cc:1467] Processing CreateTablet for tablet 3e58f8d2231b4e2d900ec099ae700d09 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260810 18:51:06.778628 16601 tablet_service.cc:1467] Processing CreateTablet for tablet c3b33c2c321842498975ee1086991616 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260810 18:51:06.779524 16486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c87160448924498ac7de68947b3dc19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.779815 16732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e58f8d2231b4e2d900ec099ae700d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.779933 16601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b33c2c321842498975ee1086991616. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.783383 16478 tablet_service.cc:1467] Processing CreateTablet for tablet 91b6af1458cd44d8ba26b43579613ecb (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260810 18:51:06.784605 16478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91b6af1458cd44d8ba26b43579613ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.789175 16728 tablet_service.cc:1467] Processing CreateTablet for tablet 2deb399ca7cb48198e6d3910bef67202 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260810 18:51:06.769521 16817 tablet_bootstrap.cc:654] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:06.773226 16600 tablet_service.cc:1467] Processing CreateTablet for tablet 2deb399ca7cb48198e6d3910bef67202 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260810 18:51:06.809092 16731 tablet_service.cc:1467] Processing CreateTablet for tablet c3b33c2c321842498975ee1086991616 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260810 18:51:06.809886 16600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2deb399ca7cb48198e6d3910bef67202. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.810527 16731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b33c2c321842498975ee1086991616. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.812662 16728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2deb399ca7cb48198e6d3910bef67202. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.775381 16483 tablet_service.cc:1467] Processing CreateTablet for tablet 8547cf266dd24ff289d7d8ff5e2174d0 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260810 18:51:06.817584 16483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8547cf266dd24ff289d7d8ff5e2174d0. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 18:51:06.819316 16415 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260810 18:51:06.824079 16735 tablet_service.cc:1467] Processing CreateTablet for tablet 02553ba44ccf4839ae0c25e0b3a1b71b (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260810 18:51:06.825340 16735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02553ba44ccf4839ae0c25e0b3a1b71b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.792128 16493 tablet_service.cc:1467] Processing CreateTablet for tablet 3e58f8d2231b4e2d900ec099ae700d09 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260810 18:51:06.767863 16482 tablet_service.cc:1467] Processing CreateTablet for tablet d16638c4ef6148f8a1e6ff4e81ae955d (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260810 18:51:06.768133 16477 tablet_service.cc:1467] Processing CreateTablet for tablet ef3f6c02785c4a76ba768b20f41f2dcd (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260810 18:51:06.826413 16493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e58f8d2231b4e2d900ec099ae700d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.826984 16482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d16638c4ef6148f8a1e6ff4e81ae955d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.798524 16818 tablet_bootstrap.cc:492] T 32567b0896594c1186a56d27e36ca0a0 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:06.800565 16814 tablet_bootstrap.cc:492] T 42bb15fb54124b8db9b482e3f17fc87c P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:06.808040 16609 tablet_service.cc:1467] Processing CreateTablet for tablet cdfc652dd163415b94d3f0bc3931a07f (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260810 18:51:06.830828 16818 ts_tablet_manager.cc:1397] T 32567b0896594c1186a56d27e36ca0a0 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.179s user 0.017s sys 0.006s
I20260810 18:51:06.830885 16814 ts_tablet_manager.cc:1397] T 42bb15fb54124b8db9b482e3f17fc87c P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.243s user 0.015s sys 0.008s
I20260810 18:51:06.831182 16609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdfc652dd163415b94d3f0bc3931a07f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.830837 16477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef3f6c02785c4a76ba768b20f41f2dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.833778 16814 raft_consensus.cc:348] T 42bb15fb54124b8db9b482e3f17fc87c P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:06.834551 16814 raft_consensus.cc:374] T 42bb15fb54124b8db9b482e3f17fc87c P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:06.834852 16814 raft_consensus.cc:729] T 42bb15fb54124b8db9b482e3f17fc87c P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:06.840848 16814 consensus_queue.cc:260] T 42bb15fb54124b8db9b482e3f17fc87c P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:06.841993 16619 tablet_service.cc:1467] Processing CreateTablet for tablet 02553ba44ccf4839ae0c25e0b3a1b71b (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260810 18:51:06.843361 16619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02553ba44ccf4839ae0c25e0b3a1b71b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.843964 16818 raft_consensus.cc:348] T 32567b0896594c1186a56d27e36ca0a0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:06.844692 16818 raft_consensus.cc:374] T 32567b0896594c1186a56d27e36ca0a0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:06.844974 16818 raft_consensus.cc:729] T 32567b0896594c1186a56d27e36ca0a0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:06.845747 16818 consensus_queue.cc:260] T 32567b0896594c1186a56d27e36ca0a0 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:06.849419 16733 tablet_service.cc:1467] Processing CreateTablet for tablet b4248aab7d8b4ed0b7c298cad63430f8 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260810 18:51:06.849777 16601 tablet_service.cc:1467] Processing CreateTablet for tablet b4248aab7d8b4ed0b7c298cad63430f8 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260810 18:51:06.850769 16733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4248aab7d8b4ed0b7c298cad63430f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.850997 16601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4248aab7d8b4ed0b7c298cad63430f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.847342 16726 tablet_service.cc:1467] Processing CreateTablet for tablet cdfc652dd163415b94d3f0bc3931a07f (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260810 18:51:06.865917 16726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdfc652dd163415b94d3f0bc3931a07f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.880738 16494 tablet_service.cc:1467] Processing CreateTablet for tablet 2deb399ca7cb48198e6d3910bef67202 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260810 18:51:06.882066 16494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2deb399ca7cb48198e6d3910bef67202. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.886826 16486 tablet_service.cc:1467] Processing CreateTablet for tablet 02553ba44ccf4839ae0c25e0b3a1b71b (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260810 18:51:06.887993 16477 tablet_service.cc:1467] Processing CreateTablet for tablet cdfc652dd163415b94d3f0bc3931a07f (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260810 18:51:06.885291 16482 tablet_service.cc:1467] Processing CreateTablet for tablet c3b33c2c321842498975ee1086991616 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260810 18:51:06.887969 16818 ts_tablet_manager.cc:1428] T 32567b0896594c1186a56d27e36ca0a0 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.057s user 0.007s sys 0.007s
I20260810 18:51:06.888979 16476 tablet_service.cc:1467] Processing CreateTablet for tablet b4248aab7d8b4ed0b7c298cad63430f8 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260810 18:51:06.892100 16477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdfc652dd163415b94d3f0bc3931a07f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.892650 16476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4248aab7d8b4ed0b7c298cad63430f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.895537 16486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02553ba44ccf4839ae0c25e0b3a1b71b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.896921 16614 tablet_service.cc:1467] Processing CreateTablet for tablet c758b1e1139941a9a8c4e703cc058061 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260810 18:51:06.898319 16614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c758b1e1139941a9a8c4e703cc058061. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.909500 16546 heartbeater.cc:503] Master 127.14.32.254:46351 was elected leader, sending a full tablet report...
I20260810 18:51:06.914151 16482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b33c2c321842498975ee1086991616. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.914494 16818 tablet_bootstrap.cc:492] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:06.920482 16814 ts_tablet_manager.cc:1428] T 42bb15fb54124b8db9b482e3f17fc87c P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.089s user 0.011s sys 0.010s
I20260810 18:51:06.920573 16730 tablet_service.cc:1467] Processing CreateTablet for tablet 7a93f1f1e1cc4e038b5ab9280d80cb23 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260810 18:51:06.922582 16604 tablet_service.cc:1467] Processing CreateTablet for tablet dcd14258d5d24692a3377387cb5c204e (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260810 18:51:06.922763 16600 tablet_service.cc:1467] Processing CreateTablet for tablet 7a93f1f1e1cc4e038b5ab9280d80cb23 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260810 18:51:06.928386 16672 heartbeater.cc:503] Master 127.14.32.254:46351 was elected leader, sending a full tablet report...
I20260810 18:51:06.929289 16611 tablet_service.cc:1467] Processing CreateTablet for tablet 2e86171b8963411f8ae8789b7b1f76f8 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260810 18:51:06.930539 16486 tablet_service.cc:1467] Processing CreateTablet for tablet c758b1e1139941a9a8c4e703cc058061 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260810 18:51:06.932698 16729 tablet_service.cc:1467] Processing CreateTablet for tablet 2e86171b8963411f8ae8789b7b1f76f8 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260810 18:51:06.933897 16729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e86171b8963411f8ae8789b7b1f76f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.936985 16817 tablet_bootstrap.cc:492] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:06.937440 16817 ts_tablet_manager.cc:1397] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.192s user 0.014s sys 0.004s
I20260810 18:51:06.937889 16726 tablet_service.cc:1467] Processing CreateTablet for tablet dcd14258d5d24692a3377387cb5c204e (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260810 18:51:06.939111 16726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcd14258d5d24692a3377387cb5c204e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.943584 16477 tablet_service.cc:1467] Processing CreateTablet for tablet 7a93f1f1e1cc4e038b5ab9280d80cb23 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260810 18:51:06.946497 16477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a93f1f1e1cc4e038b5ab9280d80cb23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.949935 16599 tablet_service.cc:1467] Processing CreateTablet for tablet e6768c6bbdc345eeb86337a837cc1d22 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260810 18:51:06.951601 16605 tablet_service.cc:1467] Processing CreateTablet for tablet 7152ca0ee8db485ab76fb9e272bbb59d (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260810 18:51:06.954432 16736 tablet_service.cc:1467] Processing CreateTablet for tablet e6768c6bbdc345eeb86337a837cc1d22 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260810 18:51:06.964596 16486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c758b1e1139941a9a8c4e703cc058061. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.967788 16611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e86171b8963411f8ae8789b7b1f76f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.968135 16736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6768c6bbdc345eeb86337a837cc1d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.983421 16604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcd14258d5d24692a3377387cb5c204e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.985883 16599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6768c6bbdc345eeb86337a837cc1d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.989177 16818 tablet_bootstrap.cc:654] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:06.991374 16605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7152ca0ee8db485ab76fb9e272bbb59d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.049364 16600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a93f1f1e1cc4e038b5ab9280d80cb23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.993054 16492 tablet_service.cc:1467] Processing CreateTablet for tablet e6768c6bbdc345eeb86337a837cc1d22 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260810 18:51:07.050472 16492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6768c6bbdc345eeb86337a837cc1d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.000758 16817 raft_consensus.cc:348] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:07.005213 16728 tablet_service.cc:1467] Processing CreateTablet for tablet 7152ca0ee8db485ab76fb9e272bbb59d (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260810 18:51:07.059806 16817 raft_consensus.cc:374] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:07.060145 16817 raft_consensus.cc:729] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:07.061235 16817 consensus_queue.cc:260] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:07.007879 16609 tablet_service.cc:1467] Processing CreateTablet for tablet 6943a9cdcb8f48dd817a90f357bf54f1 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260810 18:51:07.012524 16610 tablet_service.cc:1467] Processing CreateTablet for tablet 6a1d591f794a4601bf3c950c2c57b3cf (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260810 18:51:07.011209 16477 tablet_service.cc:1467] Processing CreateTablet for tablet 7152ca0ee8db485ab76fb9e272bbb59d (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260810 18:51:07.024514 16824 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:07.024196 16614 tablet_service.cc:1467] Processing CreateTablet for tablet ab9362a1df854c4fad0204ee2f9f78ad (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260810 18:51:07.031447 16732 tablet_service.cc:1467] Processing CreateTablet for tablet c758b1e1139941a9a8c4e703cc058061 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260810 18:51:07.063026 16482 tablet_service.cc:1467] Processing CreateTablet for tablet ab9362a1df854c4fad0204ee2f9f78ad (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260810 18:51:07.034821 16739 tablet_service.cc:1467] Processing CreateTablet for tablet 6943a9cdcb8f48dd817a90f357bf54f1 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260810 18:51:07.038038 16599 tablet_service.cc:1467] Processing CreateTablet for tablet 4e623fc5613147efbb77f01c3b0b7e81 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260810 18:51:07.039286 16475 tablet_service.cc:1467] Processing CreateTablet for tablet 2e86171b8963411f8ae8789b7b1f76f8 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260810 18:51:07.040279 16733 tablet_service.cc:1467] Processing CreateTablet for tablet 6a1d591f794a4601bf3c950c2c57b3cf (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260810 18:51:06.968003 16483 tablet_service.cc:1467] Processing CreateTablet for tablet dcd14258d5d24692a3377387cb5c204e (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260810 18:51:07.064105 16731 tablet_service.cc:1467] Processing CreateTablet for tablet ab9362a1df854c4fad0204ee2f9f78ad (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260810 18:51:07.065032 16824 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:07.063987 16609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6943a9cdcb8f48dd817a90f357bf54f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.075421 16735 tablet_service.cc:1467] Processing CreateTablet for tablet 4e623fc5613147efbb77f01c3b0b7e81 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260810 18:51:07.082157 16824 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:07.082865 16599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e623fc5613147efbb77f01c3b0b7e81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.048180 16478 tablet_service.cc:1467] Processing CreateTablet for tablet 6a1d591f794a4601bf3c950c2c57b3cf (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260810 18:51:07.034476 16730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a93f1f1e1cc4e038b5ab9280d80cb23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.064805 16477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7152ca0ee8db485ab76fb9e272bbb59d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:06.978930 16814 tablet_bootstrap.cc:492] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:07.090373 16475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e86171b8963411f8ae8789b7b1f76f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.090845 16604 tablet_service.cc:1467] Processing CreateTablet for tablet fabdebbbbc4b4fda8760bb69a90a9454 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260810 18:51:07.091418 16483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcd14258d5d24692a3377387cb5c204e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.099275 16614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab9362a1df854c4fad0204ee2f9f78ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.115376 16482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab9362a1df854c4fad0204ee2f9f78ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.115374 16610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a1d591f794a4601bf3c950c2c57b3cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.119335 16731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab9362a1df854c4fad0204ee2f9f78ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.120150 16732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c758b1e1139941a9a8c4e703cc058061. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.123080 16478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a1d591f794a4601bf3c950c2c57b3cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.125057 16604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fabdebbbbc4b4fda8760bb69a90a9454. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.125484 16817 ts_tablet_manager.cc:1428] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.188s user 0.009s sys 0.004s
I20260810 18:51:07.126222 16787 heartbeater.cc:503] Master 127.14.32.254:46351 was elected leader, sending a full tablet report...
I20260810 18:51:07.126484 16817 tablet_bootstrap.cc:492] T ccee0891d8df4f81beb2cfbe1b5485f2 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:07.128497 16619 tablet_service.cc:1467] Processing CreateTablet for tablet f95820d00c5148f2bc4169d3aafe28dd (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260810 18:51:07.129771 16619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f95820d00c5148f2bc4169d3aafe28dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.133265 16601 tablet_service.cc:1467] Processing CreateTablet for tablet 156c7477728b4962a7069618cdf12b92 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260810 18:51:07.134464 16601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156c7477728b4962a7069618cdf12b92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.137099 16476 tablet_service.cc:1467] Processing CreateTablet for tablet 6943a9cdcb8f48dd817a90f357bf54f1 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260810 18:51:07.138384 16476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6943a9cdcb8f48dd817a90f357bf54f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.139248 16733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a1d591f794a4601bf3c950c2c57b3cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.141654 16735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e623fc5613147efbb77f01c3b0b7e81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.143260 16728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7152ca0ee8db485ab76fb9e272bbb59d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.148650 16486 tablet_service.cc:1467] Processing CreateTablet for tablet fabdebbbbc4b4fda8760bb69a90a9454 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260810 18:51:07.150259 16486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fabdebbbbc4b4fda8760bb69a90a9454. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.151366 16739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6943a9cdcb8f48dd817a90f357bf54f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.159910 16493 tablet_service.cc:1467] Processing CreateTablet for tablet 4e623fc5613147efbb77f01c3b0b7e81 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260810 18:51:07.161221 16493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e623fc5613147efbb77f01c3b0b7e81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.193002 16611 tablet_service.cc:1467] Processing CreateTablet for tablet 192ad39135ed48d5adab628ba706791a (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260810 18:51:07.194657 16611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192ad39135ed48d5adab628ba706791a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.200160 16817 tablet_bootstrap.cc:654] T ccee0891d8df4f81beb2cfbe1b5485f2 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:07.201974 16600 tablet_service.cc:1467] Processing CreateTablet for tablet 1cb6043f3f9f4b38b58e608973ac019c (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260810 18:51:07.205739 16736 tablet_service.cc:1467] Processing CreateTablet for tablet f95820d00c5148f2bc4169d3aafe28dd (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260810 18:51:07.207013 16736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f95820d00c5148f2bc4169d3aafe28dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.209841 16599 tablet_service.cc:1467] Processing CreateTablet for tablet 8cbe74806d8349e690b392275de70292 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260810 18:51:07.223690 16477 tablet_service.cc:1467] Processing CreateTablet for tablet 156c7477728b4962a7069618cdf12b92 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260810 18:51:07.225304 16477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156c7477728b4962a7069618cdf12b92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.233767 16814 tablet_bootstrap.cc:654] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:07.240957 16831 raft_consensus.cc:482] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:07.241453 16831 raft_consensus.cc:504] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:07.255379 16600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cb6043f3f9f4b38b58e608973ac019c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.222879 16730 tablet_service.cc:1467] Processing CreateTablet for tablet 156c7477728b4962a7069618cdf12b92 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260810 18:51:07.231902 16729 tablet_service.cc:1467] Processing CreateTablet for tablet fabdebbbbc4b4fda8760bb69a90a9454 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260810 18:51:07.274569 16494 tablet_service.cc:1467] Processing CreateTablet for tablet f95820d00c5148f2bc4169d3aafe28dd (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260810 18:51:07.275878 16494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f95820d00c5148f2bc4169d3aafe28dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.275661 16614 tablet_service.cc:1467] Processing CreateTablet for tablet 04ec62e59fcf42ddaf65f9e6b21b49e6 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260810 18:51:07.274569 16619 tablet_service.cc:1467] Processing CreateTablet for tablet 7ef7c65533724e678b75830002906e2c (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260810 18:51:07.283716 16729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fabdebbbbc4b4fda8760bb69a90a9454. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.286305 16826 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:07.286759 16826 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:07.289206 16826 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:07.304790 16599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cbe74806d8349e690b392275de70292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.314260 16609 tablet_service.cc:1467] Processing CreateTablet for tablet 681439c754d942b7847d73744abbe269 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260810 18:51:07.331010 16739 tablet_service.cc:1467] Processing CreateTablet for tablet 1cb6043f3f9f4b38b58e608973ac019c (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260810 18:51:07.306974 16831 leader_election.cc:288] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:07.346069 16475 tablet_service.cc:1467] Processing CreateTablet for tablet 1cb6043f3f9f4b38b58e608973ac019c (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260810 18:51:07.347385 16475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cb6043f3f9f4b38b58e608973ac019c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.359508 16818 tablet_bootstrap.cc:492] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:07.359258 16476 tablet_service.cc:1467] Processing CreateTablet for tablet 8cbe74806d8349e690b392275de70292 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260810 18:51:07.359970 16818 ts_tablet_manager.cc:1397] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.449s user 0.019s sys 0.020s
I20260810 18:51:07.360522 16476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cbe74806d8349e690b392275de70292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.362713 16818 raft_consensus.cc:348] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:51:07.363883 16818 raft_consensus.cc:374] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:07.364187 16818 raft_consensus.cc:729] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:07.364823 16818 consensus_queue.cc:260] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:51:07.367808 16605 tablet_service.cc:1467] Processing CreateTablet for tablet 0c3053ea9475438bab46cba64f4cc2fa (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260810 18:51:07.347374 16492 tablet_service.cc:1467] Processing CreateTablet for tablet 192ad39135ed48d5adab628ba706791a (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260810 18:51:07.370563 16492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192ad39135ed48d5adab628ba706791a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.371027 16604 tablet_service.cc:1467] Processing CreateTablet for tablet 33bce85658374382881d7f8d510b407d (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260810 18:51:07.399261 16733 tablet_service.cc:1467] Processing CreateTablet for tablet 192ad39135ed48d5adab628ba706791a (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260810 18:51:07.399389 16735 tablet_service.cc:1467] Processing CreateTablet for tablet 8cbe74806d8349e690b392275de70292 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260810 18:51:07.403159 16614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04ec62e59fcf42ddaf65f9e6b21b49e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.404170 16731 tablet_service.cc:1467] Processing CreateTablet for tablet 04ec62e59fcf42ddaf65f9e6b21b49e6 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260810 18:51:07.411900 16610 tablet_service.cc:1467] Processing CreateTablet for tablet bfc23c22a6cb41a5a57a107ebbed91af (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260810 18:51:07.418084 16732 tablet_service.cc:1467] Processing CreateTablet for tablet 0c3053ea9475438bab46cba64f4cc2fa (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260810 18:51:07.416487 16729 tablet_service.cc:1467] Processing CreateTablet for tablet 681439c754d942b7847d73744abbe269 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260810 18:51:07.339670 16730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156c7477728b4962a7069618cdf12b92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.401535 16726 tablet_service.cc:1467] Processing CreateTablet for tablet 7ef7c65533724e678b75830002906e2c (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260810 18:51:07.372310 16817 tablet_bootstrap.cc:492] T ccee0891d8df4f81beb2cfbe1b5485f2 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:07.371160 16818 ts_tablet_manager.cc:1428] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20260810 18:51:07.425585 16619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ef7c65533724e678b75830002906e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.440035 16818 tablet_bootstrap.cc:492] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:07.434573 16817 ts_tablet_manager.cc:1397] T ccee0891d8df4f81beb2cfbe1b5485f2 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.308s user 0.007s sys 0.014s
I20260810 18:51:07.440990 16735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cbe74806d8349e690b392275de70292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.441692 16475 tablet_service.cc:1467] Processing CreateTablet for tablet 681439c754d942b7847d73744abbe269 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260810 18:51:07.442880 16739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cb6043f3f9f4b38b58e608973ac019c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.443491 16605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c3053ea9475438bab46cba64f4cc2fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.451361 16604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33bce85658374382881d7f8d510b407d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.451496 16818 tablet_bootstrap.cc:654] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:07.455405 16733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192ad39135ed48d5adab628ba706791a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.456184 16732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c3053ea9475438bab46cba64f4cc2fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.463666 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:51:07.465431 16573 leader_election.cc:341] T 42bb15fb54124b8db9b482e3f17fc87c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:07.443421 16483 tablet_service.cc:1467] Processing CreateTablet for tablet 04ec62e59fcf42ddaf65f9e6b21b49e6 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260810 18:51:07.442993 16475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 681439c754d942b7847d73744abbe269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.479404 16478 tablet_service.cc:1467] Processing CreateTablet for tablet bfc23c22a6cb41a5a57a107ebbed91af (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260810 18:51:07.456670 16609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 681439c754d942b7847d73744abbe269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.480453 16729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 681439c754d942b7847d73744abbe269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.483429 16483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04ec62e59fcf42ddaf65f9e6b21b49e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.493757 16482 tablet_service.cc:1467] Processing CreateTablet for tablet f642e61a7e3c45de883c78d5e5c49712 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260810 18:51:07.494707 16726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ef7c65533724e678b75830002906e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.495527 16610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfc23c22a6cb41a5a57a107ebbed91af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.499394 16731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04ec62e59fcf42ddaf65f9e6b21b49e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.499691 16486 tablet_service.cc:1467] Processing CreateTablet for tablet 33bce85658374382881d7f8d510b407d (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260810 18:51:07.455853 16494 tablet_service.cc:1467] Processing CreateTablet for tablet 7ef7c65533724e678b75830002906e2c (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260810 18:51:07.500706 16814 tablet_bootstrap.cc:492] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:07.484769 16728 tablet_service.cc:1467] Processing CreateTablet for tablet 33bce85658374382881d7f8d510b407d (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260810 18:51:07.501220 16814 ts_tablet_manager.cc:1397] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.529s user 0.014s sys 0.018s
I20260810 18:51:07.501946 16728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33bce85658374382881d7f8d510b407d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.505949 16817 raft_consensus.cc:348] T ccee0891d8df4f81beb2cfbe1b5485f2 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:07.506748 16817 raft_consensus.cc:374] T ccee0891d8df4f81beb2cfbe1b5485f2 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:07.507057 16817 raft_consensus.cc:729] T ccee0891d8df4f81beb2cfbe1b5485f2 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:07.517163 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:07.506062 16601 tablet_service.cc:1467] Processing CreateTablet for tablet 694fa0ce1cd246a985705792780adee0 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260810 18:51:07.495016 16482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f642e61a7e3c45de883c78d5e5c49712. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.522086 16478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfc23c22a6cb41a5a57a107ebbed91af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.522758 16817 consensus_queue.cc:260] T ccee0891d8df4f81beb2cfbe1b5485f2 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:07.530157 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:07.533775 16604 tablet_service.cc:1467] Processing CreateTablet for tablet abdb4e808ce3454ab9a52051b369e5d0 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260810 18:51:07.471002 16611 tablet_service.cc:1467] Processing CreateTablet for tablet f642e61a7e3c45de883c78d5e5c49712 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260810 18:51:07.559834 16611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f642e61a7e3c45de883c78d5e5c49712. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.550585 16619 tablet_service.cc:1467] Processing CreateTablet for tablet 7dd134517a994065a45633249a3ce405 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260810 18:51:07.560204 16492 tablet_service.cc:1467] Processing CreateTablet for tablet abdb4e808ce3454ab9a52051b369e5d0 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260810 18:51:07.560935 16619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dd134517a994065a45633249a3ce405. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 18:51:07.562407 16448 leader_election.cc:341] T 32567b0896594c1186a56d27e36ca0a0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:07.543783 16493 tablet_service.cc:1467] Processing CreateTablet for tablet 0c3053ea9475438bab46cba64f4cc2fa (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260810 18:51:07.545781 16814 raft_consensus.cc:348] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:51:07.563932 16814 raft_consensus.cc:374] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:07.564333 16814 raft_consensus.cc:729] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:07.565265 16814 consensus_queue.cc:260] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:51:07.551427 16826 raft_consensus.cc:482] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:07.551196 16610 tablet_service.cc:1467] Processing CreateTablet for tablet f250ae6ba6aa4e55b02199717a2bcde3 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260810 18:51:07.557672 16477 tablet_service.cc:1467] Processing CreateTablet for tablet 694fa0ce1cd246a985705792780adee0 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260810 18:51:07.566706 16826 raft_consensus.cc:504] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:51:07.567898 16610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f250ae6ba6aa4e55b02199717a2bcde3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.568713 16826 leader_election.cc:288] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:07.571461 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:07.572049 16636 raft_consensus.cc:2454] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 0.
I20260810 18:51:07.573158 16448 leader_election.cc:302] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:07.573980 16826 raft_consensus.cc:2781] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:07.574304 16826 raft_consensus.cc:482] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:07.574617 16826 raft_consensus.cc:3037] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:07.558180 16605 tablet_service.cc:1467] Processing CreateTablet for tablet 649d43610ba24875a4418e39b25126ba (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260810 18:51:07.575791 16605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 649d43610ba24875a4418e39b25126ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.559456 16736 tablet_service.cc:1467] Processing CreateTablet for tablet bfc23c22a6cb41a5a57a107ebbed91af (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260810 18:51:07.579792 16736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfc23c22a6cb41a5a57a107ebbed91af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.547372 16486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33bce85658374382881d7f8d510b407d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.586830 16739 tablet_service.cc:1467] Processing CreateTablet for tablet f642e61a7e3c45de883c78d5e5c49712 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
W20260810 18:51:07.589449 16575 leader_election.cc:341] T 42bb15fb54124b8db9b482e3f17fc87c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:07.589244 16729 tablet_service.cc:1467] Processing CreateTablet for tablet 694fa0ce1cd246a985705792780adee0 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260810 18:51:07.590027 16575 leader_election.cc:302] T 42bb15fb54124b8db9b482e3f17fc87c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:07.590852 16824 raft_consensus.cc:2726] T 42bb15fb54124b8db9b482e3f17fc87c P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 18:51:07.591029 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 59522.2us
I20260810 18:51:07.603772 16739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f642e61a7e3c45de883c78d5e5c49712. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.611471 16494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ef7c65533724e678b75830002906e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.600389 16814 ts_tablet_manager.cc:1428] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.099s user 0.010s sys 0.000s
I20260810 18:51:07.612154 16492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdb4e808ce3454ab9a52051b369e5d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.594388 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:07.612880 16477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 694fa0ce1cd246a985705792780adee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.596500 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:07.596933 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:07.615381 16814 tablet_bootstrap.cc:492] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:07.615362 16493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c3053ea9475438bab46cba64f4cc2fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.617667 16818 tablet_bootstrap.cc:492] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:07.618147 16818 ts_tablet_manager.cc:1397] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.178s user 0.023s sys 0.022s
I20260810 18:51:07.601819 16817 ts_tablet_manager.cc:1428] T ccee0891d8df4f81beb2cfbe1b5485f2 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.157s user 0.005s sys 0.001s
I20260810 18:51:07.619966 16817 tablet_bootstrap.cc:492] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:07.599871 16604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdb4e808ce3454ab9a52051b369e5d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.606125 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:07.623593 16695 leader_election.cc:341] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
W20260810 18:51:07.624590 16450 leader_election.cc:341] T 32567b0896594c1186a56d27e36ca0a0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:07.625025 16450 leader_election.cc:302] T 32567b0896594c1186a56d27e36ca0a0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:07.625604 16729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 694fa0ce1cd246a985705792780adee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.626422 16483 tablet_service.cc:1467] Processing CreateTablet for tablet 649d43610ba24875a4418e39b25126ba (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260810 18:51:07.627715 16483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 649d43610ba24875a4418e39b25126ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.604621 16826 raft_consensus.cc:504] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:51:07.635327 16601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 694fa0ce1cd246a985705792780adee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.602216 16731 tablet_service.cc:1467] Processing CreateTablet for tablet f250ae6ba6aa4e55b02199717a2bcde3 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260810 18:51:07.637817 16731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f250ae6ba6aa4e55b02199717a2bcde3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.641376 16818 raft_consensus.cc:348] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:07.642163 16818 raft_consensus.cc:374] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:07.642463 16818 raft_consensus.cc:729] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:07.643445 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:07.644021 16636 raft_consensus.cc:3037] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:07.649173 16609 tablet_service.cc:1467] Processing CreateTablet for tablet 3f3efea4006f43ee933f50d3e328a1f0 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260810 18:51:07.594620 16735 tablet_service.cc:1467] Processing CreateTablet for tablet abdb4e808ce3454ab9a52051b369e5d0 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260810 18:51:07.650475 16609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f3efea4006f43ee933f50d3e328a1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.650641 16636 raft_consensus.cc:2454] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:07.651268 16735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdb4e808ce3454ab9a52051b369e5d0. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 18:51:07.651639 16701 leader_election.cc:341] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:07.652086 16701 leader_election.cc:302] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:07.653296 16831 raft_consensus.cc:2726] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:07.654300 16826 leader_election.cc:288] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:07.654831 16826 raft_consensus.cc:2726] T 32567b0896594c1186a56d27e36ca0a0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:07.655329 16600 tablet_service.cc:1467] Processing CreateTablet for tablet da1fa324d17c479383225a8cea494a80 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260810 18:51:07.660339 16600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da1fa324d17c479383225a8cea494a80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.661726 16732 tablet_service.cc:1467] Processing CreateTablet for tablet 649d43610ba24875a4418e39b25126ba (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260810 18:51:07.665019 16728 tablet_service.cc:1467] Processing CreateTablet for tablet 8810b6b0b676405daf5ffddc50a4d21f (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260810 18:51:07.596603 16476 tablet_service.cc:1467] Processing CreateTablet for tablet f250ae6ba6aa4e55b02199717a2bcde3 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260810 18:51:07.666045 16448 leader_election.cc:302] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:07.666769 16476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f250ae6ba6aa4e55b02199717a2bcde3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.666911 16826 raft_consensus.cc:2781] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:07.667435 16826 raft_consensus.cc:686] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:07.668247 16826 consensus_queue.cc:237] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:51:07.673501 16475 tablet_service.cc:1467] Processing CreateTablet for tablet 8810b6b0b676405daf5ffddc50a4d21f (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260810 18:51:07.674810 16475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8810b6b0b676405daf5ffddc50a4d21f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.666810 16733 tablet_service.cc:1467] Processing CreateTablet for tablet aa5164e185b84d0a9bf72af61bf78f84 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260810 18:51:07.676060 16486 tablet_service.cc:1467] Processing CreateTablet for tablet aa5164e185b84d0a9bf72af61bf78f84 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260810 18:51:07.677457 16817 tablet_bootstrap.cc:654] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:07.658452 16818 consensus_queue.cc:260] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:07.642657 16610 tablet_service.cc:1467] Processing CreateTablet for tablet aa5164e185b84d0a9bf72af61bf78f84 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260810 18:51:07.685648 16733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa5164e185b84d0a9bf72af61bf78f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.693742 16726 tablet_service.cc:1467] Processing CreateTablet for tablet 7dd134517a994065a45633249a3ce405 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260810 18:51:07.694902 16732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 649d43610ba24875a4418e39b25126ba. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 18:51:07.701000 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 76549.1us
W20260810 18:51:07.701782 16450 leader_election.cc:341] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:07.704391 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
W20260810 18:51:07.708901 16450 leader_election.cc:341] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:07.711994 16486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa5164e185b84d0a9bf72af61bf78f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.711042 16726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dd134517a994065a45633249a3ce405. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.711499 16478 tablet_service.cc:1467] Processing CreateTablet for tablet 7dd134517a994065a45633249a3ce405 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260810 18:51:07.713260 16728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8810b6b0b676405daf5ffddc50a4d21f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.713565 16478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dd134517a994065a45633249a3ce405. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.722868 16814 tablet_bootstrap.cc:654] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:07.680388 16614 tablet_service.cc:1467] Processing CreateTablet for tablet 8810b6b0b676405daf5ffddc50a4d21f (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260810 18:51:07.745497 16605 tablet_service.cc:1467] Processing CreateTablet for tablet 867cf3e179de4be2acd5230d0a3e0ebb (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260810 18:51:07.753108 16493 tablet_service.cc:1467] Processing CreateTablet for tablet 867cf3e179de4be2acd5230d0a3e0ebb (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260810 18:51:07.765663 16605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 867cf3e179de4be2acd5230d0a3e0ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.785326 16730 tablet_service.cc:1467] Processing CreateTablet for tablet da1fa324d17c479383225a8cea494a80 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260810 18:51:07.786444 16736 tablet_service.cc:1467] Processing CreateTablet for tablet 867cf3e179de4be2acd5230d0a3e0ebb (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260810 18:51:07.786923 16494 tablet_service.cc:1467] Processing CreateTablet for tablet da1fa324d17c479383225a8cea494a80 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260810 18:51:07.788636 16739 tablet_service.cc:1467] Processing CreateTablet for tablet d39f8d52fbd04fc58cb3c20d457bab3d (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260810 18:51:07.718662 16492 tablet_service.cc:1467] Processing CreateTablet for tablet 3f3efea4006f43ee933f50d3e328a1f0 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260810 18:51:07.816648 16730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da1fa324d17c479383225a8cea494a80. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 18:51:07.825296 16415 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260810 18:51:07.829093 16614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8810b6b0b676405daf5ffddc50a4d21f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.841629 16599 tablet_service.cc:1467] Processing CreateTablet for tablet d39f8d52fbd04fc58cb3c20d457bab3d (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260810 18:51:07.854250 16599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d39f8d52fbd04fc58cb3c20d457bab3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.854385 16817 tablet_bootstrap.cc:492] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:07.854897 16817 ts_tablet_manager.cc:1397] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.235s user 0.018s sys 0.013s
I20260810 18:51:07.838622 16494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da1fa324d17c479383225a8cea494a80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.859669 16492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f3efea4006f43ee933f50d3e328a1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.859956 16344 catalog_manager.cc:5495] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 0 to 1, leader changed from <none> to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:07.864108 16818 ts_tablet_manager.cc:1428] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.246s user 0.046s sys 0.013s
I20260810 18:51:07.858628 16477 tablet_service.cc:1467] Processing CreateTablet for tablet d39f8d52fbd04fc58cb3c20d457bab3d (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260810 18:51:07.865134 16818 tablet_bootstrap.cc:492] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:07.865483 16477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d39f8d52fbd04fc58cb3c20d457bab3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.866595 16493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 867cf3e179de4be2acd5230d0a3e0ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.871613 16817 raft_consensus.cc:348] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:07.873881 16817 raft_consensus.cc:374] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:07.874347 16817 raft_consensus.cc:729] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:07.875245 16610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa5164e185b84d0a9bf72af61bf78f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.875564 16817 consensus_queue.cc:260] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:07.878532 16817 ts_tablet_manager.cc:1428] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20260810 18:51:07.887795 16736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 867cf3e179de4be2acd5230d0a3e0ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.890266 16739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d39f8d52fbd04fc58cb3c20d457bab3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.840678 16729 tablet_service.cc:1467] Processing CreateTablet for tablet 3f3efea4006f43ee933f50d3e328a1f0 (DEFAULT_TABLE table=test-workload [id=5aa4952cbee8447ca410249e65af5af6]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260810 18:51:07.891700 16729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f3efea4006f43ee933f50d3e328a1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:07.904348 16817 tablet_bootstrap.cc:492] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:07.905081 16814 tablet_bootstrap.cc:492] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:07.905535 16814 ts_tablet_manager.cc:1397] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.290s user 0.019s sys 0.013s
I20260810 18:51:07.909220 16814 raft_consensus.cc:348] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:07.910079 16814 raft_consensus.cc:374] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:07.910548 16817 tablet_bootstrap.cc:654] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:07.919692 16818 tablet_bootstrap.cc:654] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:07.910449 16814 raft_consensus.cc:729] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:07.930819 16831 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:07.931393 16831 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:07.933691 16831 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:07.934809 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:07.935307 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:07.937217 16695 leader_election.cc:341] T ccee0891d8df4f81beb2cfbe1b5485f2 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
W20260810 18:51:07.937842 16701 leader_election.cc:341] T ccee0891d8df4f81beb2cfbe1b5485f2 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:07.938318 16701 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:07.939910 16831 raft_consensus.cc:2726] T ccee0891d8df4f81beb2cfbe1b5485f2 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:07.943529 16814 consensus_queue.cc:260] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:07.949971 16788 tablet.cc:2369] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:51:07.955821 16814 ts_tablet_manager.cc:1428] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.050s user 0.007s sys 0.001s
I20260810 18:51:07.957290 16814 tablet_bootstrap.cc:492] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:07.976251 16831 raft_consensus.cc:482] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:07.979344 16831 raft_consensus.cc:504] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:07.981323 16831 leader_election.cc:288] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:07.982640 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:07.984311 16701 leader_election.cc:341] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:07.991598 16817 tablet_bootstrap.cc:492] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:07.992143 16817 ts_tablet_manager.cc:1397] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.088s user 0.024s sys 0.037s
I20260810 18:51:07.997071 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:51:07.998701 16695 leader_election.cc:341] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:07.999154 16695 leader_election.cc:302] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:07.999465 16817 raft_consensus.cc:348] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.000206 16814 tablet_bootstrap.cc:654] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.000293 16831 raft_consensus.cc:2726] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:08.002544 16818 tablet_bootstrap.cc:492] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:08.003037 16818 ts_tablet_manager.cc:1397] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.138s user 0.012s sys 0.016s
I20260810 18:51:08.000401 16817 raft_consensus.cc:374] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.004367 16817 raft_consensus.cc:729] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.005137 16817 consensus_queue.cc:260] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.007769 16817 ts_tablet_manager.cc:1428] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20260810 18:51:08.009567 16818 raft_consensus.cc:348] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.010349 16818 raft_consensus.cc:374] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.010632 16818 raft_consensus.cc:729] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.011420 16818 consensus_queue.cc:260] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.013113 16817 tablet_bootstrap.cc:492] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:08.014326 16818 ts_tablet_manager.cc:1428] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.011s user 0.003s sys 0.002s
I20260810 18:51:08.015314 16818 tablet_bootstrap.cc:492] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:08.021406 16818 tablet_bootstrap.cc:654] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.022238 16817 tablet_bootstrap.cc:654] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.027760 16814 tablet_bootstrap.cc:492] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:08.028290 16814 ts_tablet_manager.cc:1397] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.071s user 0.013s sys 0.000s
I20260810 18:51:08.031332 16814 raft_consensus.cc:348] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.031991 16814 raft_consensus.cc:374] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.032271 16814 raft_consensus.cc:729] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.032989 16814 consensus_queue.cc:260] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.034312 16818 tablet_bootstrap.cc:492] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:08.034780 16818 ts_tablet_manager.cc:1397] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.009s
I20260810 18:51:08.035414 16814 ts_tablet_manager.cc:1428] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 18:51:08.037461 16818 raft_consensus.cc:348] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.040169 16814 tablet_bootstrap.cc:492] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:08.038203 16818 raft_consensus.cc:374] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.041213 16818 raft_consensus.cc:729] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.042027 16818 consensus_queue.cc:260] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.044539 16818 ts_tablet_manager.cc:1428] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.009s user 0.002s sys 0.005s
I20260810 18:51:08.044726 16817 tablet_bootstrap.cc:492] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:08.045239 16817 ts_tablet_manager.cc:1397] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.003s
I20260810 18:51:08.048223 16817 raft_consensus.cc:348] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.049064 16817 raft_consensus.cc:374] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.049381 16817 raft_consensus.cc:729] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.050179 16817 consensus_queue.cc:260] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.051402 16814 tablet_bootstrap.cc:654] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.046700 16818 tablet_bootstrap.cc:492] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:08.054817 16817 ts_tablet_manager.cc:1428] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20260810 18:51:08.055945 16817 tablet_bootstrap.cc:492] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:08.061208 16814 tablet_bootstrap.cc:492] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:08.061708 16818 tablet_bootstrap.cc:654] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.061806 16814 ts_tablet_manager.cc:1397] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.004s
I20260810 18:51:08.065302 16817 tablet_bootstrap.cc:654] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.064808 16814 raft_consensus.cc:348] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.065846 16814 raft_consensus.cc:374] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.066231 16814 raft_consensus.cc:729] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.067272 16814 consensus_queue.cc:260] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.069658 16814 ts_tablet_manager.cc:1428] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260810 18:51:08.070055 16818 tablet_bootstrap.cc:492] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:08.070629 16818 ts_tablet_manager.cc:1397] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.008s
I20260810 18:51:08.071038 16814 tablet_bootstrap.cc:492] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:08.071698 16826 raft_consensus.cc:482] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:08.072165 16826 raft_consensus.cc:504] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.073140 16817 tablet_bootstrap.cc:492] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:08.073553 16817 ts_tablet_manager.cc:1397] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.005s
I20260810 18:51:08.074246 16826 leader_election.cc:288] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:08.075795 16817 raft_consensus.cc:348] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:51:08.076261 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:08.076557 16817 raft_consensus.cc:374] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.076887 16817 raft_consensus.cc:729] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.077934 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:08.079176 16450 leader_election.cc:341] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
W20260810 18:51:08.079704 16448 leader_election.cc:341] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:08.078583 16818 raft_consensus.cc:348] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.080147 16818 raft_consensus.cc:374] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.080121 16448 leader_election.cc:302] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:08.080636 16814 tablet_bootstrap.cc:654] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.080564 16818 raft_consensus.cc:729] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.081589 16826 raft_consensus.cc:2726] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:08.081394 16818 consensus_queue.cc:260] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.077680 16817 consensus_queue.cc:260] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:51:08.083417 16831 raft_consensus.cc:482] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:08.083658 16818 ts_tablet_manager.cc:1428] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260810 18:51:08.083868 16831 raft_consensus.cc:504] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:51:08.084862 16818 tablet_bootstrap.cc:492] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:08.085783 16831 leader_election.cc:288] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:08.086717 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:08.090200 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:08.091190 16818 tablet_bootstrap.cc:654] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.090992 16636 raft_consensus.cc:2379] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 1: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:08.095459 16701 leader_election.cc:302] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:51:08.100370 16450 consensus_peers.cc:487] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea -> Peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Couldn't send request to peer 980f343dd1fd4e049143f8b9d9f2f1be. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:08.101245 16831 raft_consensus.cc:3037] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.112771 16831 raft_consensus.cc:2726] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:08.115248 16814 tablet_bootstrap.cc:492] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:08.116255 16814 ts_tablet_manager.cc:1397] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.045s user 0.013s sys 0.011s
I20260810 18:51:08.119834 16814 raft_consensus.cc:348] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.120626 16814 raft_consensus.cc:374] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.120872 16814 raft_consensus.cc:729] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.121027 16817 ts_tablet_manager.cc:1428] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.047s user 0.009s sys 0.018s
I20260810 18:51:08.121610 16814 consensus_queue.cc:260] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.122146 16817 tablet_bootstrap.cc:492] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:08.122385 16818 tablet_bootstrap.cc:492] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:08.122824 16818 ts_tablet_manager.cc:1397] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.038s user 0.017s sys 0.016s
I20260810 18:51:08.123881 16814 ts_tablet_manager.cc:1428] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 18:51:08.124754 16814 tablet_bootstrap.cc:492] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:08.125353 16818 raft_consensus.cc:348] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.126083 16818 raft_consensus.cc:374] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.126348 16818 raft_consensus.cc:729] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.127122 16818 consensus_queue.cc:260] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.128077 16817 tablet_bootstrap.cc:654] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.131207 16814 tablet_bootstrap.cc:654] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.133239 16818 ts_tablet_manager.cc:1428] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260810 18:51:08.134434 16818 tablet_bootstrap.cc:492] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:08.139310 16817 tablet_bootstrap.cc:492] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:08.139798 16817 ts_tablet_manager.cc:1397] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.012s
I20260810 18:51:08.142413 16818 tablet_bootstrap.cc:654] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.142416 16817 raft_consensus.cc:348] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.143410 16817 raft_consensus.cc:374] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.143724 16817 raft_consensus.cc:729] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.145637 16817 consensus_queue.cc:260] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.147267 16814 tablet_bootstrap.cc:492] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:08.147884 16814 ts_tablet_manager.cc:1397] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.008s
I20260810 18:51:08.147925 16817 ts_tablet_manager.cc:1428] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260810 18:51:08.148639 16826 raft_consensus.cc:482] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:08.149041 16817 tablet_bootstrap.cc:492] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:08.149086 16826 raft_consensus.cc:504] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.151557 16814 raft_consensus.cc:348] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.152488 16814 raft_consensus.cc:374] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.152338 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:08.152832 16814 raft_consensus.cc:729] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.153375 16826 leader_election.cc:288] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:08.153877 16814 consensus_queue.cc:260] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.154902 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:08.155174 16636 raft_consensus.cc:2454] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 0.
I20260810 18:51:08.155550 16750 raft_consensus.cc:2454] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 0.
I20260810 18:51:08.156817 16450 leader_election.cc:302] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:08.157876 16826 raft_consensus.cc:2781] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:08.158231 16826 raft_consensus.cc:482] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:08.158550 16826 raft_consensus.cc:3037] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.160182 16817 tablet_bootstrap.cc:654] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.161474 16818 tablet_bootstrap.cc:492] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:08.161964 16818 ts_tablet_manager.cc:1397] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.016s
I20260810 18:51:08.163426 16824 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:08.164029 16824 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.164633 16818 raft_consensus.cc:348] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.165400 16818 raft_consensus.cc:374] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.165722 16818 raft_consensus.cc:729] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.166626 16818 consensus_queue.cc:260] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.168960 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:08.169569 16818 ts_tablet_manager.cc:1428] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 18:51:08.169788 16508 raft_consensus.cc:2454] T 32567b0896594c1186a56d27e36ca0a0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 0.
I20260810 18:51:08.170727 16818 tablet_bootstrap.cc:492] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:08.170717 16824 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:08.171924 16573 leader_election.cc:302] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:08.173785 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:08.174619 16750 raft_consensus.cc:2454] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 0.
I20260810 18:51:08.182682 16818 tablet_bootstrap.cc:654] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.182868 16824 raft_consensus.cc:2781] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:08.183578 16824 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:08.184082 16824 raft_consensus.cc:3037] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.195362 16818 tablet_bootstrap.cc:492] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:08.195950 16818 ts_tablet_manager.cc:1397] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.025s user 0.018s sys 0.001s
I20260810 18:51:08.199028 16818 raft_consensus.cc:348] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.199947 16818 raft_consensus.cc:374] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.200280 16818 raft_consensus.cc:729] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.201301 16818 consensus_queue.cc:260] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.208046 16818 ts_tablet_manager.cc:1428] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.012s user 0.003s sys 0.005s
I20260810 18:51:08.209043 16818 tablet_bootstrap.cc:492] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:08.215481 16818 tablet_bootstrap.cc:654] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.220345 16826 raft_consensus.cc:504] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.225986 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:08.226784 16636 raft_consensus.cc:3037] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.235836 16855 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:08.236508 16855 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.238201 16636 raft_consensus.cc:2454] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:08.241176 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:08.242035 16750 raft_consensus.cc:2454] T ccee0891d8df4f81beb2cfbe1b5485f2 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 0.
I20260810 18:51:08.243692 16448 leader_election.cc:302] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:08.245278 16854 raft_consensus.cc:2781] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:08.248947 16855 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:08.250330 16450 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:08.251860 16855 raft_consensus.cc:2781] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:08.252806 16855 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:08.253733 16855 raft_consensus.cc:3037] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.260838 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:08.263199 16636 raft_consensus.cc:2454] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 0.
I20260810 18:51:08.271610 16854 raft_consensus.cc:686] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:08.272949 16854 consensus_queue.cc:237] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.280429 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:08.281210 16750 raft_consensus.cc:3037] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.287201 16826 leader_election.cc:288] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:08.291239 16824 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.295454 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:08.296223 16508 raft_consensus.cc:3037] T 32567b0896594c1186a56d27e36ca0a0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.313170 16508 raft_consensus.cc:2454] T 32567b0896594c1186a56d27e36ca0a0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:08.332165 16907 consensus_queue.cc:1035] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:08.334708 16750 raft_consensus.cc:2454] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:08.336841 16902 raft_consensus.cc:482] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:08.339757 16573 leader_election.cc:302] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:08.342904 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:08.349155 16889 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:08.349296 16752 raft_consensus.cc:3037] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.349647 16889 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.337565 16855 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.357031 16814 ts_tablet_manager.cc:1428] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.209s user 0.049s sys 0.097s
I20260810 18:51:08.357896 16889 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:08.358106 16814 tablet_bootstrap.cc:492] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:08.347720 16824 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:08.360366 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:08.361039 16508 raft_consensus.cc:2454] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 0.
I20260810 18:51:08.362952 16695 leader_election.cc:302] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:08.361055 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:08.365731 16814 tablet_bootstrap.cc:654] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.352895 16902 raft_consensus.cc:504] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.367471 16633 raft_consensus.cc:2454] T 42bb15fb54124b8db9b482e3f17fc87c P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 0.
I20260810 18:51:08.367195 16889 raft_consensus.cc:2781] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:08.368417 16889 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:08.368772 16889 raft_consensus.cc:3037] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.370182 16902 leader_election.cc:288] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:08.370769 16902 raft_consensus.cc:2781] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:08.370795 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:08.372028 16902 raft_consensus.cc:686] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:08.372844 16508 raft_consensus.cc:2454] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 0.
I20260810 18:51:08.370563 16343 catalog_manager.cc:5495] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 0 to 1, leader changed from <none> to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:08.376596 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:08.377404 16749 raft_consensus.cc:3037] T ccee0891d8df4f81beb2cfbe1b5485f2 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.373771 16817 tablet_bootstrap.cc:492] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:08.379348 16817 ts_tablet_manager.cc:1397] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.230s user 0.092s sys 0.092s
I20260810 18:51:08.380290 16889 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.382009 16817 raft_consensus.cc:348] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.382676 16817 raft_consensus.cc:374] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.382925 16817 raft_consensus.cc:729] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.383128 16855 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:08.374234 16818 tablet_bootstrap.cc:492] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:08.375502 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:08.376225 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:08.383610 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:08.384594 16633 raft_consensus.cc:3037] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.384831 16507 raft_consensus.cc:3037] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.383577 16817 consensus_queue.cc:260] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.376740 16752 raft_consensus.cc:2454] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:08.385655 16818 ts_tablet_manager.cc:1397] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.177s user 0.041s sys 0.100s
I20260810 18:51:08.385178 16573 leader_election.cc:302] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:08.386246 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:08.394652 16633 raft_consensus.cc:2454] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:08.394872 16824 raft_consensus.cc:2781] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:08.395241 16632 raft_consensus.cc:3037] T 42bb15fb54124b8db9b482e3f17fc87c P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.396091 16750 raft_consensus.cc:2454] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 0.
I20260810 18:51:08.395820 16818 raft_consensus.cc:348] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.398111 16817 ts_tablet_manager.cc:1428] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20260810 18:51:08.406330 16817 tablet_bootstrap.cc:492] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:08.409310 16632 raft_consensus.cc:2454] T 42bb15fb54124b8db9b482e3f17fc87c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:08.394151 16507 raft_consensus.cc:2454] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:08.410605 16701 leader_election.cc:302] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:08.399331 16749 raft_consensus.cc:2454] T ccee0891d8df4f81beb2cfbe1b5485f2 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:08.400082 16902 consensus_queue.cc:237] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.402146 16448 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:08.384090 16889 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:08.404443 16824 raft_consensus.cc:482] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:08.405311 16818 raft_consensus.cc:374] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.411473 16831 raft_consensus.cc:2781] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:08.412599 16824 raft_consensus.cc:3037] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.413045 16831 raft_consensus.cc:686] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:08.414362 16831 consensus_queue.cc:237] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.413259 16855 raft_consensus.cc:2781] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:08.413358 16818 raft_consensus.cc:729] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.420113 16818 consensus_queue.cc:260] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.420876 16824 raft_consensus.cc:504] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.422914 16818 ts_tablet_manager.cc:1428] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.036s user 0.005s sys 0.001s
I20260810 18:51:08.424058 16818 tablet_bootstrap.cc:492] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:08.425414 16814 tablet_bootstrap.cc:492] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:08.421293 16855 raft_consensus.cc:686] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:08.425887 16814 ts_tablet_manager.cc:1397] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.068s user 0.009s sys 0.000s
I20260810 18:51:08.426396 16855 consensus_queue.cc:237] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.428814 16814 raft_consensus.cc:348] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.427361 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:08.429823 16814 raft_consensus.cc:374] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.430259 16814 raft_consensus.cc:729] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.431116 16814 consensus_queue.cc:260] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.433506 16814 ts_tablet_manager.cc:1428] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 18:51:08.434612 16814 tablet_bootstrap.cc:492] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:08.434832 16826 raft_consensus.cc:482] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:08.435510 16826 raft_consensus.cc:504] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.438072 16826 leader_election.cc:288] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:08.415752 16817 tablet_bootstrap.cc:654] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.440670 16814 tablet_bootstrap.cc:654] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.440147 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:08.442359 16824 leader_election.cc:288] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:51:08.442442 16450 leader_election.cc:341] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:08.443527 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:08.443750 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:08.444296 16749 raft_consensus.cc:3037] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.444543 16507 raft_consensus.cc:3037] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.447311 16818 tablet_bootstrap.cc:654] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
W20260810 18:51:08.451202 16448 leader_election.cc:341] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:08.451637 16448 leader_election.cc:302] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:08.452783 16826 raft_consensus.cc:2726] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:08.454605 16814 tablet_bootstrap.cc:492] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:08.455147 16814 ts_tablet_manager.cc:1397] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.006s
I20260810 18:51:08.458384 16814 raft_consensus.cc:348] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.459321 16814 raft_consensus.cc:374] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.459647 16814 raft_consensus.cc:729] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.457664 16749 raft_consensus.cc:2454] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:08.460513 16814 consensus_queue.cc:260] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.461784 16575 leader_election.cc:302] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:08.463114 16824 raft_consensus.cc:2781] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:08.463582 16824 raft_consensus.cc:686] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:08.464782 16814 ts_tablet_manager.cc:1428] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260810 18:51:08.464362 16824 consensus_queue.cc:237] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.457221 16507 raft_consensus.cc:2454] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:08.468492 16345 catalog_manager.cc:5495] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 0 to 1, leader changed from <none> to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:51:08.471232 16817 tablet_bootstrap.cc:492] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:08.471715 16817 ts_tablet_manager.cc:1397] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.066s user 0.010s sys 0.000s
I20260810 18:51:08.474654 16817 raft_consensus.cc:348] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.475250 16814 tablet_bootstrap.cc:492] T 9e960488563344af83b979c260c2dd06 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:08.475534 16817 raft_consensus.cc:374] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.475893 16817 raft_consensus.cc:729] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.476733 16817 consensus_queue.cc:260] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.479177 16817 ts_tablet_manager.cc:1428] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 18:51:08.480192 16817 tablet_bootstrap.cc:492] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:08.482820 16814 tablet_bootstrap.cc:654] T 9e960488563344af83b979c260c2dd06 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.487425 16817 tablet_bootstrap.cc:654] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.484797 16343 catalog_manager.cc:5495] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:08.492834 16818 tablet_bootstrap.cc:492] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:08.493254 16818 ts_tablet_manager.cc:1397] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.069s user 0.007s sys 0.017s
I20260810 18:51:08.497169 16345 catalog_manager.cc:5495] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 0 to 1, leader changed from <none> to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:08.497625 16818 raft_consensus.cc:348] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.498541 16818 raft_consensus.cc:374] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.498844 16818 raft_consensus.cc:729] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.499588 16818 consensus_queue.cc:260] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.502086 16814 tablet_bootstrap.cc:492] T 9e960488563344af83b979c260c2dd06 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:08.502259 16818 ts_tablet_manager.cc:1428] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.009s user 0.006s sys 0.002s
I20260810 18:51:08.502485 16814 ts_tablet_manager.cc:1397] T 9e960488563344af83b979c260c2dd06 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.000s
I20260810 18:51:08.503144 16818 tablet_bootstrap.cc:492] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:08.506650 16814 raft_consensus.cc:348] T 9e960488563344af83b979c260c2dd06 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.507347 16814 raft_consensus.cc:374] T 9e960488563344af83b979c260c2dd06 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.507601 16814 raft_consensus.cc:729] T 9e960488563344af83b979c260c2dd06 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.508818 16818 tablet_bootstrap.cc:654] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.508680 16814 consensus_queue.cc:260] T 9e960488563344af83b979c260c2dd06 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.511372 16814 ts_tablet_manager.cc:1428] T 9e960488563344af83b979c260c2dd06 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260810 18:51:08.511878 16855 raft_consensus.cc:482] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:08.512478 16814 tablet_bootstrap.cc:492] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:08.512319 16855 raft_consensus.cc:504] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.514883 16855 leader_election.cc:288] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:08.516822 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:08.517436 16636 raft_consensus.cc:2454] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 0.
I20260810 18:51:08.517657 16817 tablet_bootstrap.cc:492] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:08.518106 16817 ts_tablet_manager.cc:1397] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.004s
I20260810 18:51:08.518244 16814 tablet_bootstrap.cc:654] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.518610 16448 leader_election.cc:302] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:08.519455 16855 raft_consensus.cc:2781] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:08.519804 16855 raft_consensus.cc:482] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:08.520144 16855 raft_consensus.cc:3037] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.520962 16817 raft_consensus.cc:348] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.521739 16817 raft_consensus.cc:374] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.522068 16817 raft_consensus.cc:729] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.522744 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:08.523514 16749 raft_consensus.cc:2454] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 0.
I20260810 18:51:08.523363 16817 consensus_queue.cc:260] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.525461 16817 ts_tablet_manager.cc:1428] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 18:51:08.526679 16817 tablet_bootstrap.cc:492] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:08.537405 16855 raft_consensus.cc:504] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.541580 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:08.542150 16636 raft_consensus.cc:3037] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.544919 16817 tablet_bootstrap.cc:654] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.545288 16814 tablet_bootstrap.cc:492] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:08.545722 16814 ts_tablet_manager.cc:1397] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.033s user 0.014s sys 0.013s
I20260810 18:51:08.544656 16338 catalog_manager.cc:5495] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:08.548235 16814 raft_consensus.cc:348] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.548907 16814 raft_consensus.cc:374] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.548938 16636 raft_consensus.cc:2454] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:08.549266 16814 raft_consensus.cc:729] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.550040 16448 leader_election.cc:302] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:08.549980 16814 consensus_queue.cc:260] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.550822 16907 raft_consensus.cc:2781] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:08.551270 16907 raft_consensus.cc:686] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:08.552362 16814 ts_tablet_manager.cc:1428] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260810 18:51:08.552042 16907 consensus_queue.cc:237] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.553186 16814 tablet_bootstrap.cc:492] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:08.558991 16818 tablet_bootstrap.cc:492] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:08.559448 16818 ts_tablet_manager.cc:1397] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.056s user 0.010s sys 0.000s
I20260810 18:51:08.560191 16814 tablet_bootstrap.cc:654] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.562041 16818 raft_consensus.cc:348] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.563534 16824 raft_consensus.cc:482] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:08.571753 16814 tablet_bootstrap.cc:492] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:08.574329 16824 raft_consensus.cc:504] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.575043 16814 ts_tablet_manager.cc:1397] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.004s
I20260810 18:51:08.576385 16824 leader_election.cc:288] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:08.578043 16814 raft_consensus.cc:348] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.578254 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:08.578589 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:08.578791 16814 raft_consensus.cc:374] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.579170 16814 raft_consensus.cc:729] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
W20260810 18:51:08.579952 16575 leader_election.cc:341] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:08.579871 16814 consensus_queue.cc:260] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.580549 16818 raft_consensus.cc:374] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.581073 16907 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:08.581508 16907 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.582237 16814 ts_tablet_manager.cc:1428] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20260810 18:51:08.583117 16814 tablet_bootstrap.cc:492] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:08.584029 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:08.584630 16636 raft_consensus.cc:2454] T 9e960488563344af83b979c260c2dd06 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 0.
W20260810 18:51:08.584976 16573 leader_election.cc:341] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:08.585569 16573 leader_election.cc:302] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:08.585716 16448 leader_election.cc:302] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:08.586616 16826 raft_consensus.cc:2781] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:08.586989 16826 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:08.587538 16826 raft_consensus.cc:3037] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.588459 16814 tablet_bootstrap.cc:654] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.586934 16824 raft_consensus.cc:2726] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:08.577394 16344 catalog_manager.cc:5495] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 0 to 1, leader changed from <none> to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:08.596997 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:08.597801 16749 raft_consensus.cc:2454] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 0.
I20260810 18:51:08.598253 16826 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.599282 16907 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:08.580822 16818 raft_consensus.cc:729] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.601936 16818 consensus_queue.cc:260] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.604933 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:08.605494 16749 raft_consensus.cc:3037] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.605623 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:08.606166 16636 raft_consensus.cc:3037] T 9e960488563344af83b979c260c2dd06 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.609283 16826 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:08.610282 16818 ts_tablet_manager.cc:1428] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.051s user 0.006s sys 0.000s
I20260810 18:51:08.611595 16818 tablet_bootstrap.cc:492] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:08.616915 16824 raft_consensus.cc:482] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:08.617525 16749 raft_consensus.cc:2454] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:08.617364 16824 raft_consensus.cc:504] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.618736 16450 leader_election.cc:302] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:08.619374 16824 leader_election.cc:288] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:08.620555 16826 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:08.620412 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:08.620903 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:08.623108 16814 tablet_bootstrap.cc:492] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
W20260810 18:51:08.623312 16573 leader_election.cc:341] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:08.624397 16889 raft_consensus.cc:482] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:08.625315 16814 ts_tablet_manager.cc:1397] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.042s user 0.012s sys 0.008s
W20260810 18:51:08.625403 16575 leader_election.cc:341] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:08.621608 16826 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.624401 16817 tablet_bootstrap.cc:492] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:08.627110 16817 ts_tablet_manager.cc:1397] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.101s user 0.034s sys 0.039s
I20260810 18:51:08.627745 16575 leader_election.cc:302] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:08.628360 16814 raft_consensus.cc:348] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.628651 16826 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:08.625344 16889 raft_consensus.cc:504] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.629215 16814 raft_consensus.cc:374] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.621790 16907 raft_consensus.cc:2781] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:08.629542 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:08.630033 16817 raft_consensus.cc:348] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.630551 16814 raft_consensus.cc:729] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.630724 16817 raft_consensus.cc:374] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.631196 16907 raft_consensus.cc:686] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:08.631428 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:08.631337 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:51:08.631268 16448 leader_election.cc:341] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:08.631909 16817 raft_consensus.cc:729] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.632115 16750 raft_consensus.cc:3037] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.632488 16907 consensus_queue.cc:237] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:08.632958 16450 leader_election.cc:341] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:08.632675 16817 consensus_queue.cc:260] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.633371 16450 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:08.634595 16826 raft_consensus.cc:2726] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:08.635426 16889 leader_election.cc:288] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:08.637287 16817 ts_tablet_manager.cc:1428] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.010s user 0.005s sys 0.003s
I20260810 18:51:08.637185 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:08.637702 16814 consensus_queue.cc:260] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.638484 16507 raft_consensus.cc:2454] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 0.
I20260810 18:51:08.639133 16817 tablet_bootstrap.cc:492] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:08.639323 16902 raft_consensus.cc:2726] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:08.640148 16814 ts_tablet_manager.cc:1428] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20260810 18:51:08.641175 16695 leader_election.cc:302] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:08.644191 16817 tablet_bootstrap.cc:654] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.645879 16814 tablet_bootstrap.cc:492] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:08.622545 16831 raft_consensus.cc:482] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:08.646588 16831 raft_consensus.cc:504] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.648634 16831 leader_election.cc:288] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:08.649308 16831 raft_consensus.cc:2781] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:08.649708 16831 raft_consensus.cc:482] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:08.653440 16831 raft_consensus.cc:3037] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.655493 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:08.656154 16507 raft_consensus.cc:2454] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 0.
I20260810 18:51:08.657032 16817 tablet_bootstrap.cc:492] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:08.657233 16695 leader_election.cc:302] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:08.657593 16817 ts_tablet_manager.cc:1397] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.005s
I20260810 18:51:08.657491 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:08.651526 16814 tablet_bootstrap.cc:654] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.658170 16632 raft_consensus.cc:2454] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 0.
I20260810 18:51:08.660328 16817 raft_consensus.cc:348] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.660833 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:08.661180 16817 raft_consensus.cc:374] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.661471 16632 raft_consensus.cc:2454] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 0.
I20260810 18:51:08.661646 16817 raft_consensus.cc:729] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.662418 16889 raft_consensus.cc:2781] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:08.662535 16817 consensus_queue.cc:260] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.662883 16831 raft_consensus.cc:504] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.662883 16889 raft_consensus.cc:482] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:08.666930 16889 raft_consensus.cc:3037] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.668906 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:08.669540 16507 raft_consensus.cc:3037] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.672773 16750 raft_consensus.cc:2454] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:08.676124 16507 raft_consensus.cc:2454] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:08.677332 16695 leader_election.cc:302] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:08.677543 16889 raft_consensus.cc:504] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.678237 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:08.678781 16632 raft_consensus.cc:3037] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.680351 16818 tablet_bootstrap.cc:654] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.665156 16817 ts_tablet_manager.cc:1428] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 18:51:08.683730 16817 tablet_bootstrap.cc:492] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:08.685284 16632 raft_consensus.cc:2454] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:08.686715 16951 raft_consensus.cc:2781] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:08.688139 16831 leader_election.cc:288] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:08.621704 16855 leader_election.cc:288] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:08.689292 16817 tablet_bootstrap.cc:654] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.691918 16889 leader_election.cc:288] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:08.702986 16818 tablet_bootstrap.cc:492] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:08.694605 16814 tablet_bootstrap.cc:492] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:08.701244 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:08.728792 16818 ts_tablet_manager.cc:1397] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.117s user 0.010s sys 0.015s
I20260810 18:51:08.729082 16814 ts_tablet_manager.cc:1397] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.083s user 0.030s sys 0.009s
I20260810 18:51:08.726284 16951 raft_consensus.cc:686] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:08.729241 16507 raft_consensus.cc:3037] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.693178 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:08.730139 16633 raft_consensus.cc:3037] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.730029 16951 consensus_queue.cc:237] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.692366 16636 raft_consensus.cc:2454] T 9e960488563344af83b979c260c2dd06 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:08.731659 16818 raft_consensus.cc:348] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.732394 16818 raft_consensus.cc:374] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.732693 16818 raft_consensus.cc:729] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.733281 16818 consensus_queue.cc:260] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.734633 16958 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:08.735512 16818 ts_tablet_manager.cc:1428] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 18:51:08.735116 16958 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.736428 16818 tablet_bootstrap.cc:492] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:08.737285 16814 raft_consensus.cc:348] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.737871 16507 raft_consensus.cc:2454] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:08.739527 16695 leader_election.cc:302] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:08.740374 16889 raft_consensus.cc:2781] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:08.740826 16889 raft_consensus.cc:686] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:08.741597 16889 consensus_queue.cc:237] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.742367 16633 raft_consensus.cc:2454] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:08.742679 16958 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:08.743696 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:08.737982 16814 raft_consensus.cc:374] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.744833 16814 raft_consensus.cc:729] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.745512 16814 consensus_queue.cc:260] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.744511 16507 raft_consensus.cc:2454] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 0.
I20260810 18:51:08.748677 16814 ts_tablet_manager.cc:1428] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.019s user 0.005s sys 0.001s
I20260810 18:51:08.749833 16814 tablet_bootstrap.cc:492] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:08.753691 16817 tablet_bootstrap.cc:492] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:08.755255 16817 ts_tablet_manager.cc:1397] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.072s user 0.016s sys 0.015s
I20260810 18:51:08.755653 16814 tablet_bootstrap.cc:654] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.758405 16817 raft_consensus.cc:348] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.761171 16817 raft_consensus.cc:374] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.761353 16817 raft_consensus.cc:729] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.761746 16817 consensus_queue.cc:260] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.762859 16818 tablet_bootstrap.cc:654] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.763638 16817 ts_tablet_manager.cc:1428] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260810 18:51:08.764602 16817 tablet_bootstrap.cc:492] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:08.768018 16695 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:08.768882 16951 raft_consensus.cc:2781] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:08.769474 16951 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:08.769922 16951 raft_consensus.cc:3037] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.769941 16814 tablet_bootstrap.cc:492] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:08.772073 16814 ts_tablet_manager.cc:1397] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.005s
I20260810 18:51:08.772801 16817 tablet_bootstrap.cc:654] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.774585 16814 raft_consensus.cc:348] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.775346 16814 raft_consensus.cc:374] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.775631 16814 raft_consensus.cc:729] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.776228 16814 consensus_queue.cc:260] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.776819 16818 tablet_bootstrap.cc:492] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:08.777252 16818 ts_tablet_manager.cc:1397] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.004s
I20260810 18:51:08.778294 16814 ts_tablet_manager.cc:1428] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260810 18:51:08.779156 16814 tablet_bootstrap.cc:492] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:08.779116 16951 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.778975 16342 catalog_manager.cc:5495] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 0 to 1, leader changed from <none> to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:51:08.779654 16818 raft_consensus.cc:348] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.780992 16818 raft_consensus.cc:374] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.781262 16818 raft_consensus.cc:729] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.782071 16818 consensus_queue.cc:260] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.783980 16951 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:08.784183 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:08.784861 16507 raft_consensus.cc:3037] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.785205 16814 tablet_bootstrap.cc:654] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.786056 16817 tablet_bootstrap.cc:492] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:08.786506 16817 ts_tablet_manager.cc:1397] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.000s
I20260810 18:51:08.788934 16817 raft_consensus.cc:348] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.789588 16817 raft_consensus.cc:374] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.784190 16818 ts_tablet_manager.cc:1428] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 18:51:08.789844 16907 consensus_queue.cc:1035] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:51:08.790835 16818 tablet_bootstrap.cc:492] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:08.792060 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:08.792696 16633 raft_consensus.cc:3037] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.762441 16338 catalog_manager.cc:5495] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 0 to 1, leader changed from <none> to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:08.793942 16907 raft_consensus.cc:482] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:08.789857 16817 raft_consensus.cc:729] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.794448 16907 raft_consensus.cc:504] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.795115 16817 consensus_queue.cc:260] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.796480 16907 leader_election.cc:288] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:08.797150 16817 ts_tablet_manager.cc:1428] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260810 18:51:08.798220 16817 tablet_bootstrap.cc:492] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:08.799428 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:08.798223 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:08.801429 16631 raft_consensus.cc:2454] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 0.
W20260810 18:51:08.802405 16450 leader_election.cc:341] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:08.803846 16448 leader_election.cc:302] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:08.809645 16817 tablet_bootstrap.cc:654] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.805786 16818 tablet_bootstrap.cc:654] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.811269 16907 raft_consensus.cc:2781] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:08.813206 16907 raft_consensus.cc:482] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:08.813577 16907 raft_consensus.cc:3037] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.816111 16824 consensus_queue.cc:1035] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:08.817703 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:08.823438 16889 raft_consensus.cc:482] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:08.824023 16889 raft_consensus.cc:504] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.825265 16507 raft_consensus.cc:2454] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:08.827476 16889 leader_election.cc:288] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:08.832196 16855 consensus_queue.cc:1035] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:51:08.832573 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:08.833237 16631 raft_consensus.cc:2454] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 0.
I20260810 18:51:08.835664 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:51:08.837153 16695 leader_election.cc:341] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:08.837939 16701 leader_election.cc:302] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:08.838714 16889 raft_consensus.cc:2781] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:08.839198 16889 raft_consensus.cc:482] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:08.839655 16889 raft_consensus.cc:3037] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.840698 16855 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:08.841292 16855 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.843750 16855 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:08.845469 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:51:08.847083 16450 leader_election.cc:341] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:08.848182 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:08.849617 16448 leader_election.cc:341] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:08.849802 16818 tablet_bootstrap.cc:492] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:08.850099 16448 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:08.851019 16855 raft_consensus.cc:2726] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:08.856258 16951 raft_consensus.cc:482] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:08.856787 16951 raft_consensus.cc:504] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.858000 16817 tablet_bootstrap.cc:492] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:08.858596 16817 ts_tablet_manager.cc:1397] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.061s user 0.010s sys 0.007s
I20260810 18:51:08.859933 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:51:08.861509 16695 leader_election.cc:341] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:08.862634 16951 leader_election.cc:288] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:08.850368 16818 ts_tablet_manager.cc:1397] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.060s user 0.015s sys 0.000s
I20260810 18:51:08.863165 16817 raft_consensus.cc:348] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.864069 16817 raft_consensus.cc:374] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.864141 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:08.864454 16817 raft_consensus.cc:729] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.865028 16628 raft_consensus.cc:2454] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 0.
I20260810 18:51:08.865494 16817 consensus_queue.cc:260] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.866331 16912 consensus_queue.cc:1035] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:08.868464 16817 ts_tablet_manager.cc:1428] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20260810 18:51:08.868731 16912 raft_consensus.cc:482] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:08.869545 16817 tablet_bootstrap.cc:492] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:08.869236 16912 raft_consensus.cc:504] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.869128 16818 raft_consensus.cc:348] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.870512 16818 raft_consensus.cc:374] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:08.870874 16818 raft_consensus.cc:729] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:08.867841 16889 raft_consensus.cc:504] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.872856 16912 leader_election.cc:288] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:08.875442 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:08.876217 16506 raft_consensus.cc:2454] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 0.
I20260810 18:51:08.877058 16817 tablet_bootstrap.cc:654] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:08.878365 16573 leader_election.cc:302] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:08.879626 16912 raft_consensus.cc:2781] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:08.880085 16912 raft_consensus.cc:482] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:08.880461 16912 raft_consensus.cc:3037] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.880729 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:51:08.882241 16415 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260810 18:51:08.883450 16695 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:08.882503 16855 consensus_queue.cc:1035] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260810 18:51:08.888135 16633 raft_consensus.cc:2454] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:08.891460 16818 consensus_queue.cc:260] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.898743 16818 ts_tablet_manager.cc:1428] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.035s user 0.005s sys 0.001s
I20260810 18:51:08.880715 16701 leader_election.cc:302] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:08.894702 16951 raft_consensus.cc:2781] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:08.895282 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:08.900395 16818 tablet_bootstrap.cc:492] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:08.903689 16951 raft_consensus.cc:686] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:08.905573 16951 consensus_queue.cc:237] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.909694 16889 leader_election.cc:288] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:08.911314 16831 raft_consensus.cc:2781] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:08.915577 16831 raft_consensus.cc:482] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:08.916330 16831 raft_consensus.cc:3037] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
W20260810 18:51:08.925868 16695 leader_election.cc:341] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:08.926996 16958 consensus_queue.cc:1035] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 18:51:08.934056 16575 leader_election.cc:341] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:08.934257 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:08.935308 16633 raft_consensus.cc:3037] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.943250 16855 consensus_queue.cc:1035] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:08.920580 16338 catalog_manager.cc:5495] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 0 to 1, leader changed from <none> to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:51:08.960858 16958 consensus_queue.cc:1035] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:08.948794 16907 raft_consensus.cc:504] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.979817 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:08.980895 16636 raft_consensus.cc:3037] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:08.970430 16831 raft_consensus.cc:504] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:08.984826 16831 leader_election.cc:288] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:08.985018 16907 leader_election.cc:288] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:08.985785 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:08.989054 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:08.990084 16749 raft_consensus.cc:3037] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
W20260810 18:51:08.995792 16695 leader_election.cc:341] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:09.031193 16911 consensus_queue.cc:1035] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:09.020794 16907 raft_consensus.cc:482] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:09.051252 16831 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:09.069422 16831 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:09.056906 16907 raft_consensus.cc:504] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:09.199054 16912 raft_consensus.cc:504] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:09.206611 16818 tablet_bootstrap.cc:654] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:09.144403 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:51:09.208228 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 55018.6us
I20260810 18:51:09.171975 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:09.153959 16831 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:09.205694 16749 raft_consensus.cc:2454] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:09.226955 16817 tablet_bootstrap.cc:492] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:09.227680 16817 ts_tablet_manager.cc:1397] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.358s user 0.012s sys 0.044s
I20260810 18:51:09.223672 16907 leader_election.cc:288] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:09.230763 16814 tablet_bootstrap.cc:492] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:09.252707 16814 ts_tablet_manager.cc:1397] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.474s user 0.029s sys 0.036s
I20260810 18:51:09.215984 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:09.233177 16817 raft_consensus.cc:348] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:09.256316 16817 raft_consensus.cc:374] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:09.256695 16817 raft_consensus.cc:729] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:09.257612 16817 consensus_queue.cc:260] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:09.229198 16695 leader_election.cc:341] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
W20260810 18:51:09.272703 16701 leader_election.cc:341] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:09.260324 16628 raft_consensus.cc:3037] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:09.279961 16814 raft_consensus.cc:348] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:09.281008 16814 raft_consensus.cc:374] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:09.281472 16814 raft_consensus.cc:729] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
W20260810 18:51:09.274559 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 46658.9us
I20260810 18:51:09.286111 16701 leader_election.cc:302] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:09.288632 16831 raft_consensus.cc:2726] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:09.294858 16831 consensus_queue.cc:1035] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:09.309693 16817 ts_tablet_manager.cc:1428] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.082s user 0.008s sys 0.002s
I20260810 18:51:09.282399 16814 consensus_queue.cc:260] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:09.310891 16817 tablet_bootstrap.cc:492] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:09.317777 16814 ts_tablet_manager.cc:1428] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.065s user 0.009s sys 0.005s
I20260810 18:51:09.318816 16814 tablet_bootstrap.cc:492] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:09.306874 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:09.321050 16912 leader_election.cc:288] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:09.300459 16633 raft_consensus.cc:2454] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:09.306509 16450 leader_election.cc:302] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:51:09.349972 16990 log.cc:924] Time spent T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.072s user 0.000s sys 0.002s
I20260810 18:51:09.349874 17000 raft_consensus.cc:2781] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:09.350802 17000 raft_consensus.cc:686] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:09.352046 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:09.352696 17000 consensus_queue.cc:237] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:09.361291 16701 leader_election.cc:302] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:09.362882 16951 raft_consensus.cc:2781] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:09.363665 16951 raft_consensus.cc:686] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:09.365358 16951 consensus_queue.cc:237] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:09.366219 16831 consensus_queue.cc:1035] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:09.377017 16889 consensus_queue.cc:1035] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 18:51:09.378036 16575 leader_election.cc:341] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:09.367728 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:09.379711 16911 consensus_queue.cc:1035] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:09.368453 16907 consensus_queue.cc:1035] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260810 18:51:09.371215 16636 raft_consensus.cc:2454] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:09.381273 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:09.399761 16506 raft_consensus.cc:3037] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:09.406167 16958 consensus_queue.cc:1035] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 18:51:09.407097 16448 leader_election.cc:341] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:09.408453 16889 consensus_queue.cc:1035] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:09.411605 16855 consensus_queue.cc:1035] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 18:51:09.436803 17008 log.cc:924] Time spent T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
W20260810 18:51:09.489120 16450 leader_election.cc:341] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:09.489840 16450 leader_election.cc:302] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:09.496471 16999 raft_consensus.cc:2726] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:09.505201 16814 tablet_bootstrap.cc:654] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:09.526459 16817 tablet_bootstrap.cc:654] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:09.552006 16818 tablet_bootstrap.cc:492] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:09.552592 16818 ts_tablet_manager.cc:1397] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.653s user 0.038s sys 0.032s
I20260810 18:51:09.538648 16855 consensus_queue.cc:1035] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260810 18:51:09.566812 16506 raft_consensus.cc:2454] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:09.568126 16951 raft_consensus.cc:482] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:09.570849 16951 raft_consensus.cc:504] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:09.579172 16628 raft_consensus.cc:2454] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:09.604238 16573 leader_election.cc:302] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:51:09.650262 17020 log.cc:924] Time spent T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
W20260810 18:51:09.697844 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 83482.3us
I20260810 18:51:09.699095 16902 raft_consensus.cc:2781] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election won for term 1
W20260810 18:51:09.674438 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 114337us
I20260810 18:51:09.700193 16902 raft_consensus.cc:686] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:09.701977 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:09.701617 16902 consensus_queue.cc:237] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:09.704113 16701 leader_election.cc:302] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:09.705760 16958 raft_consensus.cc:2781] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:09.706633 16958 raft_consensus.cc:686] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
W20260810 18:51:09.706482 16701 leader_election.cc:341] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:09.708521 16958 consensus_queue.cc:237] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:09.697115 16951 leader_election.cc:288] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:09.744119 16818 raft_consensus.cc:348] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:09.720494 16911 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:09.788856 16999 consensus_queue.cc:1035] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:09.790683 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:09.767022 16911 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:09.830538 17022 log.cc:924] Time spent T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.108s user 0.002s sys 0.000s
W20260810 18:51:09.841109 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 54041.4us
I20260810 18:51:09.842794 16889 consensus_queue.cc:1035] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:09.849069 16817 tablet_bootstrap.cc:492] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:09.849630 16817 ts_tablet_manager.cc:1397] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.539s user 0.034s sys 0.069s
I20260810 18:51:09.746388 16818 raft_consensus.cc:374] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:09.851784 16818 raft_consensus.cc:729] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:09.860070 16818 consensus_queue.cc:260] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:09.869616 16817 raft_consensus.cc:348] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:09.870750 16817 raft_consensus.cc:374] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:09.871088 16817 raft_consensus.cc:729] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:09.880498 16817 consensus_queue.cc:260] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:09.882601 16818 ts_tablet_manager.cc:1428] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.330s user 0.008s sys 0.007s
I20260810 18:51:09.884130 16818 tablet_bootstrap.cc:492] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:09.893018 16817 ts_tablet_manager.cc:1428] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.043s user 0.007s sys 0.004s
I20260810 18:51:09.896162 16814 tablet_bootstrap.cc:492] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:09.896781 16814 ts_tablet_manager.cc:1397] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.578s user 0.030s sys 0.010s
W20260810 18:51:09.909343 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 50289.9us
W20260810 18:51:09.901400 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 48098.1us
I20260810 18:51:09.915584 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:51:09.917026 17023 log.cc:924] Time spent T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.121s user 0.001s sys 0.005s
I20260810 18:51:09.918795 16999 consensus_queue.cc:1035] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:51:09.920840 16817 tablet_bootstrap.cc:492] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:09.908578 16911 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:09.930642 17000 consensus_queue.cc:1035] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260810 18:51:09.953696 16814 raft_consensus.cc:348] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:09.954289 16814 raft_consensus.cc:374] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:09.954551 16814 raft_consensus.cc:729] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
W20260810 18:51:09.962373 16695 leader_election.cc:341] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:09.967470 16814 consensus_queue.cc:260] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:09.934018 16746 raft_consensus.cc:2454] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 0.
I20260810 18:51:09.980706 16506 raft_consensus.cc:3037] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:09.990419 16814 ts_tablet_manager.cc:1428] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.093s user 0.009s sys 0.001s
I20260810 18:51:09.991338 16814 tablet_bootstrap.cc:492] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:09.994761 16575 leader_election.cc:302] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:09.996603 16902 raft_consensus.cc:2781] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:09.999284 16902 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:10.000069 16902 raft_consensus.cc:3037] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:09.973927 16695 leader_election.cc:302] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:10.009215 17048 raft_consensus.cc:2726] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 18:51:10.015209 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 52946.1us
I20260810 18:51:10.025990 16817 tablet_bootstrap.cc:654] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:10.030803 17048 consensus_queue.cc:1035] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 18:51:10.042797 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 465607us
I20260810 18:51:10.045537 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:51:10.059019 17035 log.cc:924] Time spent T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.063s user 0.001s sys 0.000s
I20260810 18:51:10.060163 16818 tablet_bootstrap.cc:654] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:10.064608 16902 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.075129 17033 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:10.075887 16958 consensus_queue.cc:1035] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:10.075943 17033 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:10.070014 16573 leader_election.cc:341] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:10.084043 17033 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:10.085886 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:10.091398 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:10.099803 16902 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:10.099017 16344 catalog_manager.cc:5495] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 0 to 1, leader changed from <none> to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:51:10.101835 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:10.094151 16814 tablet_bootstrap.cc:654] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:10.104028 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:10.108614 16752 raft_consensus.cc:3037] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
W20260810 18:51:10.109649 16575 leader_election.cc:341] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
W20260810 18:51:10.111224 16573 leader_election.cc:341] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:10.107561 16506 raft_consensus.cc:2454] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 0.
I20260810 18:51:10.113732 16573 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:10.124084 17033 raft_consensus.cc:2781] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:10.134685 17033 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:10.135057 16817 tablet_bootstrap.cc:492] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:10.135174 17033 raft_consensus.cc:3037] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:10.135686 16817 ts_tablet_manager.cc:1397] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.215s user 0.015s sys 0.013s
W20260810 18:51:10.136624 16415 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260810 18:51:10.145417 16752 raft_consensus.cc:2454] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:10.149914 16575 leader_election.cc:302] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:10.157232 16817 raft_consensus.cc:348] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.158195 16817 raft_consensus.cc:374] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:10.158520 16817 raft_consensus.cc:729] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:10.159349 16817 consensus_queue.cc:260] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.151332 16912 raft_consensus.cc:2781] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:10.167783 16912 raft_consensus.cc:686] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:10.169500 16912 consensus_queue.cc:237] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.178908 16817 ts_tablet_manager.cc:1428] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.043s user 0.004s sys 0.005s
I20260810 18:51:10.187243 17048 raft_consensus.cc:482] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:10.219573 16817 tablet_bootstrap.cc:492] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:10.225047 16818 tablet_bootstrap.cc:492] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:10.225597 16818 ts_tablet_manager.cc:1397] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.342s user 0.051s sys 0.037s
I20260810 18:51:10.230610 16818 raft_consensus.cc:348] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.234217 16818 raft_consensus.cc:374] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:10.234526 16818 raft_consensus.cc:729] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:10.235118 16818 consensus_queue.cc:260] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.236688 17033 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.237613 16818 ts_tablet_manager.cc:1428] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.012s user 0.002s sys 0.003s
I20260810 18:51:10.239125 16818 tablet_bootstrap.cc:492] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:10.187911 17048 raft_consensus.cc:504] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.242693 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:10.243468 16507 raft_consensus.cc:3037] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:10.245882 16958 raft_consensus.cc:482] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:10.247334 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:10.248916 16958 raft_consensus.cc:504] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.253023 17048 leader_election.cc:288] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:10.248086 17033 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:51:10.271386 16695 consensus_peers.cc:487] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be -> Peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Couldn't send request to peer d304ac47fdf846de81337057a4ef7bea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:10.253329 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:10.263082 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:10.274325 16508 raft_consensus.cc:2454] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 0.
I20260810 18:51:10.275324 16695 leader_election.cc:302] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:10.253566 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:10.276181 17048 raft_consensus.cc:2781] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:10.276537 17048 raft_consensus.cc:482] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:10.276883 17048 raft_consensus.cc:3037] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:10.255954 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:10.278151 16575 leader_election.cc:341] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
W20260810 18:51:10.279299 16701 leader_election.cc:341] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:10.252862 16506 raft_consensus.cc:2379] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 1: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:10.283004 16814 tablet_bootstrap.cc:492] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:10.283784 16814 ts_tablet_manager.cc:1397] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.293s user 0.031s sys 0.026s
I20260810 18:51:10.283660 16695 leader_election.cc:302] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:10.262796 16958 leader_election.cc:288] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:10.284658 16889 raft_consensus.cc:3037] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:10.287338 16817 tablet_bootstrap.cc:654] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:10.286984 16507 raft_consensus.cc:2454] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:10.288671 16573 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:10.291338 17033 raft_consensus.cc:2781] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:10.294073 17033 raft_consensus.cc:686] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:10.291935 16889 raft_consensus.cc:2726] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:10.297147 16814 raft_consensus.cc:348] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.297819 16818 tablet_bootstrap.cc:654] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:10.294997 17033 consensus_queue.cc:237] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.297973 16814 raft_consensus.cc:374] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:10.298313 16814 raft_consensus.cc:729] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:10.299114 16814 consensus_queue.cc:260] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.301379 16814 ts_tablet_manager.cc:1428] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20260810 18:51:10.304929 16814 tablet_bootstrap.cc:492] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:10.319062 16814 tablet_bootstrap.cc:654] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:10.325428 17048 raft_consensus.cc:504] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.327733 16817 tablet_bootstrap.cc:492] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:10.330327 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:10.331998 16817 ts_tablet_manager.cc:1397] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.152s user 0.019s sys 0.000s
I20260810 18:51:10.332221 16507 raft_consensus.cc:3037] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:10.331364 17033 consensus_queue.cc:1035] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260810 18:51:10.349186 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:10.350682 17048 leader_election.cc:288] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:10.351039 16507 raft_consensus.cc:2454] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
W20260810 18:51:10.351293 16701 leader_election.cc:341] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:10.360103 16817 raft_consensus.cc:348] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.361205 16817 raft_consensus.cc:374] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:10.361567 16817 raft_consensus.cc:729] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:10.361210 16695 leader_election.cc:302] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:10.362437 16817 consensus_queue.cc:260] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.366114 17048 raft_consensus.cc:2781] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:10.367120 17048 raft_consensus.cc:686] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:10.368595 17048 consensus_queue.cc:237] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.376091 16817 ts_tablet_manager.cc:1428] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.044s user 0.008s sys 0.001s
I20260810 18:51:10.377318 16817 tablet_bootstrap.cc:492] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:10.392340 16889 consensus_queue.cc:1035] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:10.401475 16817 tablet_bootstrap.cc:654] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:10.422894 16818 tablet_bootstrap.cc:492] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:10.423671 16818 ts_tablet_manager.cc:1397] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.185s user 0.019s sys 0.012s
I20260810 18:51:10.430428 16814 tablet_bootstrap.cc:492] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:10.430941 16814 ts_tablet_manager.cc:1397] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.129s user 0.021s sys 0.005s
I20260810 18:51:10.431813 17033 consensus_queue.cc:1035] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260810 18:51:10.412096 16342 catalog_manager.cc:5495] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 0 to 1, leader changed from <none> to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:51:10.438865 16818 raft_consensus.cc:348] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.439455 16818 raft_consensus.cc:374] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:10.439682 16818 raft_consensus.cc:729] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:10.441642 16818 consensus_queue.cc:260] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.447229 16814 raft_consensus.cc:348] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.448199 16814 raft_consensus.cc:374] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:10.448449 16814 raft_consensus.cc:729] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:10.449128 16814 consensus_queue.cc:260] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.451749 16814 ts_tablet_manager.cc:1428] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.020s user 0.003s sys 0.003s
I20260810 18:51:10.452975 16814 tablet_bootstrap.cc:492] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:10.457661 16342 catalog_manager.cc:5495] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 0 to 1, leader changed from <none> to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } }
I20260810 18:51:10.464226 16817 tablet_bootstrap.cc:492] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:10.464711 16817 ts_tablet_manager.cc:1397] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.088s user 0.005s sys 0.016s
I20260810 18:51:10.467041 16817 raft_consensus.cc:348] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.476444 16817 raft_consensus.cc:374] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:10.477195 16817 raft_consensus.cc:729] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:10.478659 16817 consensus_queue.cc:260] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.468888 16814 tablet_bootstrap.cc:654] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:10.472792 16818 ts_tablet_manager.cc:1428] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.048s user 0.006s sys 0.000s
I20260810 18:51:10.480772 16818 tablet_bootstrap.cc:492] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:10.486943 16907 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:10.487732 16907 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.490574 16907 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:10.492096 16817 ts_tablet_manager.cc:1428] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.027s user 0.004s sys 0.003s
I20260810 18:51:10.492149 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:10.493031 16752 raft_consensus.cc:2454] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 0.
I20260810 18:51:10.492138 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:10.494773 16450 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:10.493260 16817 tablet_bootstrap.cc:492] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:10.496057 16907 raft_consensus.cc:2781] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260810 18:51:10.496281 16448 leader_election.cc:341] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:10.496603 16907 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:10.497254 16907 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:10.504318 16817 tablet_bootstrap.cc:654] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:10.508013 16907 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.511847 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:10.512921 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:10.515020 16752 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
W20260810 18:51:10.515101 16448 leader_election.cc:341] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:10.511097 16907 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:10.512632 16338 catalog_manager.cc:5495] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:10.527092 16752 raft_consensus.cc:2454] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:10.528978 16450 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:10.531358 16907 raft_consensus.cc:2781] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:10.532464 16907 raft_consensus.cc:686] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:10.510525 16818 tablet_bootstrap.cc:654] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:10.534274 16907 consensus_queue.cc:237] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.545490 16814 tablet_bootstrap.cc:492] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:10.545624 16817 tablet_bootstrap.cc:492] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:10.545966 16814 ts_tablet_manager.cc:1397] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.093s user 0.010s sys 0.002s
I20260810 18:51:10.546193 16817 ts_tablet_manager.cc:1397] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.053s user 0.009s sys 0.011s
I20260810 18:51:10.550431 16814 raft_consensus.cc:348] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.550496 16817 raft_consensus.cc:348] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.551772 16817 raft_consensus.cc:374] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:10.552105 16817 raft_consensus.cc:729] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:10.552886 16817 consensus_queue.cc:260] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.551438 16814 raft_consensus.cc:374] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:10.554014 16814 raft_consensus.cc:729] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:10.554773 16814 consensus_queue.cc:260] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.557775 16814 ts_tablet_manager.cc:1428] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.011s user 0.006s sys 0.003s
I20260810 18:51:10.558701 16814 tablet_bootstrap.cc:492] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:10.562649 16817 ts_tablet_manager.cc:1428] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.016s user 0.005s sys 0.006s
I20260810 18:51:10.563714 16817 tablet_bootstrap.cc:492] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:10.587976 16818 tablet_bootstrap.cc:492] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:10.588589 16818 ts_tablet_manager.cc:1397] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.108s user 0.016s sys 0.000s
I20260810 18:51:10.592468 16818 raft_consensus.cc:348] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.593425 16818 raft_consensus.cc:374] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:10.593762 16818 raft_consensus.cc:729] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:10.594497 16818 consensus_queue.cc:260] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.599211 16817 tablet_bootstrap.cc:654] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:10.603012 16818 ts_tablet_manager.cc:1428] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.014s user 0.009s sys 0.000s
I20260810 18:51:10.604140 16818 tablet_bootstrap.cc:492] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:10.609602 16814 tablet_bootstrap.cc:654] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:10.612540 16344 catalog_manager.cc:5495] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 0 to 1, leader changed from <none> to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:51:10.618969 16344 catalog_manager.cc:5495] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 0 to 1, leader changed from <none> to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:51:10.626497 16818 tablet_bootstrap.cc:654] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:10.627285 16817 tablet_bootstrap.cc:492] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:10.627737 16817 ts_tablet_manager.cc:1397] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.064s user 0.007s sys 0.012s
I20260810 18:51:10.628479 16814 tablet_bootstrap.cc:492] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:10.629026 16814 ts_tablet_manager.cc:1397] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.071s user 0.014s sys 0.011s
W20260810 18:51:10.629642 16573 consensus_peers.cc:487] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb -> Peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Couldn't send request to peer d304ac47fdf846de81337057a4ef7bea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:10.632989 16814 raft_consensus.cc:348] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.633733 16814 raft_consensus.cc:374] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:10.634054 16814 raft_consensus.cc:729] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:10.634865 16814 consensus_queue.cc:260] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.638517 16814 ts_tablet_manager.cc:1428] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260810 18:51:10.638603 16817 raft_consensus.cc:348] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.639372 16817 raft_consensus.cc:374] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:10.639549 16814 tablet_bootstrap.cc:492] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:10.639670 16817 raft_consensus.cc:729] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:10.640369 16817 consensus_queue.cc:260] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.642443 16817 ts_tablet_manager.cc:1428] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.014s user 0.004s sys 0.000s
I20260810 18:51:10.643177 16817 tablet_bootstrap.cc:492] T 4c87160448924498ac7de68947b3dc19 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:10.653934 16818 tablet_bootstrap.cc:492] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:10.654503 16818 ts_tablet_manager.cc:1397] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.051s user 0.002s sys 0.016s
I20260810 18:51:10.655766 16814 tablet_bootstrap.cc:654] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:10.660313 16818 raft_consensus.cc:348] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.661151 16818 raft_consensus.cc:374] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:10.661489 16818 raft_consensus.cc:729] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:10.662909 16817 tablet_bootstrap.cc:654] T 4c87160448924498ac7de68947b3dc19 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:10.662304 16818 consensus_queue.cc:260] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.665381 16818 ts_tablet_manager.cc:1428] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260810 18:51:10.666425 16818 tablet_bootstrap.cc:492] T 91b6af1458cd44d8ba26b43579613ecb P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:10.670277 17025 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:10.670892 17025 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.675661 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:51:10.677506 16695 leader_election.cc:341] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:10.684957 16912 raft_consensus.cc:482] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:10.685636 16912 raft_consensus.cc:504] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.686259 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:10.687780 16701 leader_election.cc:341] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:10.688258 16701 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:10.689224 16958 raft_consensus.cc:2726] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:10.689316 17025 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:10.689782 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:10.690925 16507 raft_consensus.cc:2379] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 1: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:51:10.692955 16912 leader_election.cc:288] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:10.695000 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:10.697400 16575 leader_election.cc:302] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:10.698400 16814 tablet_bootstrap.cc:492] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:10.698872 16912 raft_consensus.cc:3037] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:10.704311 16818 tablet_bootstrap.cc:654] T 91b6af1458cd44d8ba26b43579613ecb P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:10.698863 16814 ts_tablet_manager.cc:1397] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.060s user 0.013s sys 0.006s
I20260810 18:51:10.711728 16814 raft_consensus.cc:348] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.712604 16814 raft_consensus.cc:374] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:10.712939 16814 raft_consensus.cc:729] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:10.717818 16907 raft_consensus.cc:482] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:10.718433 16907 raft_consensus.cc:504] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.720853 16907 leader_election.cc:288] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:10.722718 16814 consensus_queue.cc:260] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.722730 16342 catalog_manager.cc:5495] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:10.726039 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:10.728014 16907 raft_consensus.cc:482] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:10.728806 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:10.729046 16907 raft_consensus.cc:504] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.730284 16814 ts_tablet_manager.cc:1428] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.023s user 0.009s sys 0.000s
I20260810 18:51:10.727331 16342 catalog_manager.cc:5495] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:10.732563 16814 tablet_bootstrap.cc:492] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:10.732151 16448 leader_election.cc:302] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:10.744755 16855 raft_consensus.cc:2726] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:10.733129 16907 leader_election.cc:288] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:10.734112 16912 raft_consensus.cc:2726] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:10.734767 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:10.736075 16818 tablet_bootstrap.cc:492] T 91b6af1458cd44d8ba26b43579613ecb P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:10.741174 16817 tablet_bootstrap.cc:492] T 4c87160448924498ac7de68947b3dc19 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:10.755295 16817 ts_tablet_manager.cc:1397] T 4c87160448924498ac7de68947b3dc19 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.112s user 0.021s sys 0.014s
I20260810 18:51:10.755306 17056 raft_consensus.cc:482] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:10.751497 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:10.757802 16818 ts_tablet_manager.cc:1397] T 91b6af1458cd44d8ba26b43579613ecb P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.092s user 0.011s sys 0.013s
I20260810 18:51:10.756786 17056 raft_consensus.cc:504] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:10.760259 16448 leader_election.cc:341] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:10.761969 16817 raft_consensus.cc:348] T 4c87160448924498ac7de68947b3dc19 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:10.763646 16450 leader_election.cc:341] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:10.764357 16450 leader_election.cc:302] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:10.762779 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:51:10.768771 16573 leader_election.cc:341] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:10.769965 17056 leader_election.cc:288] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:10.762897 16817 raft_consensus.cc:374] T 4c87160448924498ac7de68947b3dc19 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:10.770747 16817 raft_consensus.cc:729] T 4c87160448924498ac7de68947b3dc19 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:10.772747 17056 consensus_queue.cc:1035] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:10.774118 16912 raft_consensus.cc:482] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:10.771493 16817 consensus_queue.cc:260] T 4c87160448924498ac7de68947b3dc19 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.774839 16912 raft_consensus.cc:504] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.771826 16907 raft_consensus.cc:2726] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:10.779184 16338 catalog_manager.cc:5495] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 0 to 1, leader changed from <none> to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:10.785086 16818 raft_consensus.cc:348] T 91b6af1458cd44d8ba26b43579613ecb P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.785622 16818 raft_consensus.cc:374] T 91b6af1458cd44d8ba26b43579613ecb P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:10.785846 16818 raft_consensus.cc:729] T 91b6af1458cd44d8ba26b43579613ecb P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:10.789599 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:10.788549 16912 leader_election.cc:288] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:10.790860 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:10.792058 16814 tablet_bootstrap.cc:654] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
W20260810 18:51:10.792742 16573 leader_election.cc:341] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:10.793949 16817 ts_tablet_manager.cc:1428] T 4c87160448924498ac7de68947b3dc19 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.038s user 0.006s sys 0.001s
I20260810 18:51:10.795132 16817 tablet_bootstrap.cc:492] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
W20260810 18:51:10.803468 16575 leader_election.cc:341] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:10.804075 16575 leader_election.cc:302] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:10.805699 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:10.807169 16748 raft_consensus.cc:2454] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 0.
I20260810 18:51:10.808282 16912 raft_consensus.cc:2726] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:10.809548 16818 consensus_queue.cc:260] T 91b6af1458cd44d8ba26b43579613ecb P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.809986 16575 leader_election.cc:302] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:10.812471 16818 ts_tablet_manager.cc:1428] T 91b6af1458cd44d8ba26b43579613ecb P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.054s user 0.012s sys 0.000s
I20260810 18:51:10.814267 16818 tablet_bootstrap.cc:492] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:10.818929 17056 consensus_queue.cc:1035] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:10.812464 16912 raft_consensus.cc:2781] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:10.822567 16912 raft_consensus.cc:482] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:10.822961 16912 raft_consensus.cc:3037] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:10.825879 17033 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:10.826365 17033 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.830313 17033 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:10.833137 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:10.833340 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:10.834223 16748 raft_consensus.cc:2379] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 1: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:10.835364 16575 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:10.839535 17033 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:10.842222 16748 raft_consensus.cc:3037] T 4c87160448924498ac7de68947b3dc19 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:10.850450 17033 raft_consensus.cc:2726] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:10.855141 17025 consensus_queue.cc:1035] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:10.869027 16814 tablet_bootstrap.cc:492] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:10.869922 16814 ts_tablet_manager.cc:1397] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.138s user 0.009s sys 0.024s
I20260810 18:51:10.870476 16818 tablet_bootstrap.cc:654] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:10.872290 16817 tablet_bootstrap.cc:654] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:10.872711 16506 raft_consensus.cc:3037] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:10.879359 17056 consensus_queue.cc:1035] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:10.890177 16814 raft_consensus.cc:348] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.891096 16814 raft_consensus.cc:374] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:10.891456 16814 raft_consensus.cc:729] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:10.899158 16912 raft_consensus.cc:504] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.905526 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:10.906004 16814 consensus_queue.cc:260] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.906199 16912 leader_election.cc:288] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:51:10.910194 16573 leader_election.cc:341] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:10.905526 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:10.918452 16814 ts_tablet_manager.cc:1428] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.048s user 0.009s sys 0.002s
I20260810 18:51:10.919307 16814 tablet_bootstrap.cc:492] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:10.916973 16748 raft_consensus.cc:3037] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:10.936419 16814 tablet_bootstrap.cc:654] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:10.938788 17025 consensus_queue.cc:1035] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:10.945884 16817 tablet_bootstrap.cc:492] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:10.946235 16817 ts_tablet_manager.cc:1397] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.151s user 0.010s sys 0.010s
I20260810 18:51:10.950459 17116 consensus_queue.cc:1035] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260810 18:51:10.957870 16889 raft_consensus.cc:482] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:10.958386 16889 raft_consensus.cc:504] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.960331 16817 raft_consensus.cc:348] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.961144 17116 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:10.961298 16817 raft_consensus.cc:374] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:10.961710 17116 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.964254 16748 raft_consensus.cc:2454] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:10.964895 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:10.966674 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:10.967815 17025 raft_consensus.cc:482] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:10.968186 16448 leader_election.cc:302] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:10.968307 17025 raft_consensus.cc:504] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.968968 16907 raft_consensus.cc:2726] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:10.971958 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:10.972200 16575 leader_election.cc:302] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:10.972880 16508 raft_consensus.cc:2454] T 91b6af1458cd44d8ba26b43579613ecb P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 0.
I20260810 18:51:10.970919 17025 leader_election.cc:288] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:10.974170 16912 raft_consensus.cc:2781] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:10.974468 16695 leader_election.cc:302] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:10.974802 16912 raft_consensus.cc:686] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:10.975493 16818 tablet_bootstrap.cc:492] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:10.975430 16889 leader_election.cc:288] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:10.965063 17116 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:10.976320 16889 raft_consensus.cc:2781] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:10.978586 16889 raft_consensus.cc:482] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:10.976879 17033 raft_consensus.cc:482] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:10.979393 17033 raft_consensus.cc:504] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.976094 16912 consensus_queue.cc:237] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.981391 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:10.982173 16626 raft_consensus.cc:2454] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 0.
I20260810 18:51:10.982482 17033 leader_election.cc:288] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:10.982376 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:10.983349 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:10.983536 16636 raft_consensus.cc:2454] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 0.
I20260810 18:51:10.984014 16508 raft_consensus.cc:2454] T 91b6af1458cd44d8ba26b43579613ecb P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 0.
I20260810 18:51:10.961714 16817 raft_consensus.cc:729] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:10.985839 16573 leader_election.cc:302] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:10.986055 16817 consensus_queue.cc:260] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:10.988387 16817 ts_tablet_manager.cc:1428] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.042s user 0.006s sys 0.001s
I20260810 18:51:10.989169 17121 raft_consensus.cc:2781] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:10.990890 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:10.990974 17121 raft_consensus.cc:482] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:10.991833 17121 raft_consensus.cc:3037] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
W20260810 18:51:10.992285 16695 leader_election.cc:341] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:10.993402 16817 tablet_bootstrap.cc:492] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:10.976123 16818 ts_tablet_manager.cc:1397] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.162s user 0.010s sys 0.015s
I20260810 18:51:10.985208 16889 raft_consensus.cc:3037] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:11.002195 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:11.002600 16818 raft_consensus.cc:348] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.003338 16818 raft_consensus.cc:374] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:11.007835 16818 raft_consensus.cc:729] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:11.016502 16818 consensus_queue.cc:260] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.020144 16907 consensus_queue.cc:1035] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:51:11.014169 17121 raft_consensus.cc:504] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.022890 17121 leader_election.cc:288] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:11.027047 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:11.027740 16508 raft_consensus.cc:3037] T 91b6af1458cd44d8ba26b43579613ecb P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:11.029026 16701 leader_election.cc:302] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:11.029953 16814 tablet_bootstrap.cc:492] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:11.030462 16814 ts_tablet_manager.cc:1397] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.111s user 0.010s sys 0.023s
I20260810 18:51:11.030912 16507 raft_consensus.cc:3037] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:11.036006 16818 ts_tablet_manager.cc:1428] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.040s user 0.008s sys 0.001s
I20260810 18:51:11.044039 16818 tablet_bootstrap.cc:492] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:11.044005 17048 raft_consensus.cc:2781] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:11.053891 17048 raft_consensus.cc:482] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:11.054479 17048 raft_consensus.cc:3037] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:11.069011 16814 raft_consensus.cc:348] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.069927 16814 raft_consensus.cc:374] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:11.070221 16814 raft_consensus.cc:729] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:11.070914 16814 consensus_queue.cc:260] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.085274 16814 ts_tablet_manager.cc:1428] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.055s user 0.012s sys 0.000s
I20260810 18:51:11.086270 16814 tablet_bootstrap.cc:492] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:11.057091 16889 raft_consensus.cc:504] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.092612 16817 tablet_bootstrap.cc:654] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:11.091531 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:11.090662 16889 leader_election.cc:288] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:11.095760 16344 catalog_manager.cc:5495] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:11.095989 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:11.097236 16752 raft_consensus.cc:2379] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 1: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:51:11.098384 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:11.109282 16631 raft_consensus.cc:2379] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 1: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:11.099004 17025 raft_consensus.cc:482] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:11.110412 17025 raft_consensus.cc:504] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.115510 17025 leader_election.cc:288] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:11.110432 16508 raft_consensus.cc:2454] T 91b6af1458cd44d8ba26b43579613ecb P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:11.118736 16573 leader_election.cc:302] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:11.120419 17121 raft_consensus.cc:2781] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:11.121177 17121 raft_consensus.cc:686] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:11.122120 17121 consensus_queue.cc:237] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.128805 16818 tablet_bootstrap.cc:654] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:11.129844 16701 leader_election.cc:302] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:11.123597 17048 raft_consensus.cc:504] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.134804 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:11.141018 16631 raft_consensus.cc:2454] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 0.
I20260810 18:51:11.142545 17048 leader_election.cc:288] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:11.134152 16958 raft_consensus.cc:2726] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 18:51:11.143937 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:51:11.154708 16695 leader_election.cc:341] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:11.155023 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
W20260810 18:51:11.156853 16695 leader_election.cc:341] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:11.167541 16701 leader_election.cc:302] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:11.167547 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:11.168594 16631 raft_consensus.cc:3037] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:11.169994 17025 raft_consensus.cc:2781] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:11.171772 17025 raft_consensus.cc:482] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:11.172166 17025 raft_consensus.cc:3037] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:11.190493 16814 tablet_bootstrap.cc:654] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:11.191228 17121 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:11.191933 17121 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.194082 16817 tablet_bootstrap.cc:492] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:11.194664 16817 ts_tablet_manager.cc:1397] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.201s user 0.016s sys 0.027s
I20260810 18:51:11.195001 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:11.195214 16631 raft_consensus.cc:2454] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
W20260810 18:51:11.196736 16573 leader_election.cc:341] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:11.196463 17025 raft_consensus.cc:504] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.197793 17121 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:11.199298 17025 leader_election.cc:288] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:11.199541 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:11.200932 16817 raft_consensus.cc:348] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.201450 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:11.201704 16817 raft_consensus.cc:374] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:11.202036 16632 raft_consensus.cc:3037] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:11.202042 16817 raft_consensus.cc:729] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:11.203037 16817 consensus_queue.cc:260] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.201736 16701 leader_election.cc:302] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:11.205247 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:11.205626 16958 raft_consensus.cc:2781] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:11.206295 16958 raft_consensus.cc:686] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
W20260810 18:51:11.206503 16695 leader_election.cc:341] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:11.205531 17025 raft_consensus.cc:482] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:11.207396 16958 consensus_queue.cc:237] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.207726 17025 raft_consensus.cc:504] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.211012 16632 raft_consensus.cc:2454] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
W20260810 18:51:11.213027 16575 leader_election.cc:341] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:11.213312 17025 leader_election.cc:288] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:11.213621 16575 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:11.215333 17121 raft_consensus.cc:2726] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:11.216879 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:11.217825 16507 raft_consensus.cc:2454] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 0.
I20260810 18:51:11.219396 16817 ts_tablet_manager.cc:1428] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.024s user 0.005s sys 0.005s
I20260810 18:51:11.219660 16695 leader_election.cc:302] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:11.220202 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:11.220937 17143 raft_consensus.cc:2781] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:11.221525 16817 tablet_bootstrap.cc:492] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:11.221465 17143 raft_consensus.cc:482] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260810 18:51:11.222018 16701 leader_election.cc:341] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:11.222088 17143 raft_consensus.cc:3037] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
W20260810 18:51:11.219700 16415 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260810 18:51:11.229872 16818 tablet_bootstrap.cc:492] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:11.230201 17143 raft_consensus.cc:504] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.230623 16818 ts_tablet_manager.cc:1397] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.194s user 0.011s sys 0.014s
I20260810 18:51:11.247291 16818 raft_consensus.cc:348] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.248023 16818 raft_consensus.cc:374] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:11.248294 16818 raft_consensus.cc:729] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:11.250600 17143 leader_election.cc:288] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:11.253232 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:11.254220 16507 raft_consensus.cc:3037] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:11.253218 16345 catalog_manager.cc:5495] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:11.258234 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:11.258396 16701 leader_election.cc:302] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:11.259768 16818 consensus_queue.cc:260] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.263670 16958 raft_consensus.cc:2781] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:11.264330 16817 tablet_bootstrap.cc:654] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
W20260810 18:51:11.265435 16701 leader_election.cc:341] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:11.275996 16818 ts_tablet_manager.cc:1428] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.045s user 0.007s sys 0.000s
I20260810 18:51:11.277096 16818 tablet_bootstrap.cc:492] T c3b33c2c321842498975ee1086991616 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:11.279997 16814 tablet_bootstrap.cc:492] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:11.280485 16814 ts_tablet_manager.cc:1397] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.194s user 0.011s sys 0.015s
I20260810 18:51:11.279812 16507 raft_consensus.cc:2454] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:11.266525 16958 raft_consensus.cc:686] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:11.285580 16695 leader_election.cc:302] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:11.288429 17143 raft_consensus.cc:2781] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:11.290874 17143 raft_consensus.cc:686] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:11.292095 17143 consensus_queue.cc:237] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.289655 16958 consensus_queue.cc:237] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.294545 16814 raft_consensus.cc:348] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.298485 16814 raft_consensus.cc:374] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:11.298863 16814 raft_consensus.cc:729] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:11.299696 16814 consensus_queue.cc:260] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.302580 16818 tablet_bootstrap.cc:654] T c3b33c2c321842498975ee1086991616 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:11.308238 16814 ts_tablet_manager.cc:1428] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.027s user 0.008s sys 0.002s
I20260810 18:51:11.310858 16814 tablet_bootstrap.cc:492] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:11.328176 16817 tablet_bootstrap.cc:492] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:11.328719 16817 ts_tablet_manager.cc:1397] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.108s user 0.036s sys 0.027s
I20260810 18:51:11.335850 16817 raft_consensus.cc:348] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.336846 16817 raft_consensus.cc:374] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:11.337162 16817 raft_consensus.cc:729] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:11.337935 16817 consensus_queue.cc:260] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.341182 16817 ts_tablet_manager.cc:1428] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.012s user 0.009s sys 0.000s
I20260810 18:51:11.342372 16817 tablet_bootstrap.cc:492] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:11.360496 16818 tablet_bootstrap.cc:492] T c3b33c2c321842498975ee1086991616 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:11.361101 16818 ts_tablet_manager.cc:1397] T c3b33c2c321842498975ee1086991616 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.084s user 0.013s sys 0.013s
I20260810 18:51:11.362819 16817 tablet_bootstrap.cc:654] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:11.358530 16814 tablet_bootstrap.cc:654] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:11.367520 16818 raft_consensus.cc:348] T c3b33c2c321842498975ee1086991616 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.368512 16818 raft_consensus.cc:374] T c3b33c2c321842498975ee1086991616 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:11.368901 16818 raft_consensus.cc:729] T c3b33c2c321842498975ee1086991616 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:11.369722 16818 consensus_queue.cc:260] T c3b33c2c321842498975ee1086991616 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.373198 16818 ts_tablet_manager.cc:1428] T c3b33c2c321842498975ee1086991616 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.012s user 0.009s sys 0.001s
I20260810 18:51:11.375314 16818 tablet_bootstrap.cc:492] T 3e58f8d2231b4e2d900ec099ae700d09 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:11.383303 16817 tablet_bootstrap.cc:492] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:11.383834 16817 ts_tablet_manager.cc:1397] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.042s user 0.015s sys 0.004s
I20260810 18:51:11.385174 16818 tablet_bootstrap.cc:654] T 3e58f8d2231b4e2d900ec099ae700d09 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:11.388409 16814 tablet_bootstrap.cc:492] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:11.388839 16814 ts_tablet_manager.cc:1397] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.078s user 0.008s sys 0.011s
I20260810 18:51:11.389685 17033 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:11.390300 17033 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.391347 16814 raft_consensus.cc:348] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.392026 16814 raft_consensus.cc:374] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:11.392278 16814 raft_consensus.cc:729] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:11.393810 16817 raft_consensus.cc:348] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.394570 16817 raft_consensus.cc:374] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:11.394819 16817 raft_consensus.cc:729] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:11.396075 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:11.396428 17033 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:11.396723 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:11.397176 16814 consensus_queue.cc:260] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.395387 16345 catalog_manager.cc:5495] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 0 to 1, leader changed from <none> to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
W20260810 18:51:11.398553 16573 leader_election.cc:341] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:11.398904 16345 catalog_manager.cc:5495] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 0 to 1, leader changed from <none> to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:51:11.399863 16814 ts_tablet_manager.cc:1428] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.011s user 0.004s sys 0.001s
I20260810 18:51:11.395407 16817 consensus_queue.cc:260] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.400542 16345 catalog_manager.cc:5495] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 0 to 1, leader changed from <none> to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:51:11.401793 16814 tablet_bootstrap.cc:492] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:11.403127 16818 tablet_bootstrap.cc:492] T 3e58f8d2231b4e2d900ec099ae700d09 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:11.406981 16746 raft_consensus.cc:2454] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 0.
I20260810 18:51:11.412258 16817 ts_tablet_manager.cc:1428] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.028s user 0.008s sys 0.000s
I20260810 18:51:11.412392 16575 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:11.413378 16817 tablet_bootstrap.cc:492] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:11.413659 16814 tablet_bootstrap.cc:654] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:11.414650 17033 raft_consensus.cc:2781] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:11.415500 17033 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:11.416213 17033 raft_consensus.cc:3037] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:11.425194 16818 ts_tablet_manager.cc:1397] T 3e58f8d2231b4e2d900ec099ae700d09 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.042s user 0.010s sys 0.012s
I20260810 18:51:11.429128 16818 raft_consensus.cc:348] T 3e58f8d2231b4e2d900ec099ae700d09 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.436803 16818 raft_consensus.cc:374] T 3e58f8d2231b4e2d900ec099ae700d09 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:11.437156 16818 raft_consensus.cc:729] T 3e58f8d2231b4e2d900ec099ae700d09 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:11.438138 16818 consensus_queue.cc:260] T 3e58f8d2231b4e2d900ec099ae700d09 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.442453 16818 ts_tablet_manager.cc:1428] T 3e58f8d2231b4e2d900ec099ae700d09 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260810 18:51:11.443677 16818 tablet_bootstrap.cc:492] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:11.446141 16817 tablet_bootstrap.cc:654] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:11.449900 16818 tablet_bootstrap.cc:654] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:11.467988 16817 tablet_bootstrap.cc:492] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:11.467806 17116 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:11.468875 16817 ts_tablet_manager.cc:1397] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.056s user 0.007s sys 0.008s
I20260810 18:51:11.468683 17116 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.472337 16817 raft_consensus.cc:348] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.473242 16817 raft_consensus.cc:374] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:11.473665 16817 raft_consensus.cc:729] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:11.474205 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:11.474677 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:11.474524 16817 consensus_queue.cc:260] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.475042 16746 raft_consensus.cc:2454] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 0.
I20260810 18:51:11.472478 17116 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:11.477924 16450 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:11.480922 16817 ts_tablet_manager.cc:1428] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
W20260810 18:51:11.480983 16448 leader_election.cc:341] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:11.481904 16817 tablet_bootstrap.cc:492] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:11.481711 17116 raft_consensus.cc:2781] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:11.482301 17116 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:11.482818 17116 raft_consensus.cc:3037] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:11.487687 17033 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.491496 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:11.492831 16814 tablet_bootstrap.cc:492] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:11.493496 16814 ts_tablet_manager.cc:1397] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.092s user 0.018s sys 0.035s
W20260810 18:51:11.493512 16573 leader_election.cc:341] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:11.494995 17033 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:11.496814 16814 raft_consensus.cc:348] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.497815 16814 raft_consensus.cc:374] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:11.497843 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:11.498306 16814 raft_consensus.cc:729] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:11.498724 16746 raft_consensus.cc:3037] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:11.502322 16818 tablet_bootstrap.cc:492] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:11.503126 16818 ts_tablet_manager.cc:1397] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.060s user 0.025s sys 0.000s
I20260810 18:51:11.503355 16817 tablet_bootstrap.cc:654] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:11.499996 16814 consensus_queue.cc:260] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.506263 16814 ts_tablet_manager.cc:1428] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260810 18:51:11.506639 16907 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:11.507373 16814 tablet_bootstrap.cc:492] T b4248aab7d8b4ed0b7c298cad63430f8 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:11.507251 16907 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.509755 16818 raft_consensus.cc:348] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.510579 16818 raft_consensus.cc:374] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:11.510834 16818 raft_consensus.cc:729] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:11.513535 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:11.514451 16748 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 0.
I20260810 18:51:11.496745 17116 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.530107 17121 raft_consensus.cc:482] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:11.530570 16912 consensus_queue.cc:1035] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:11.530632 17121 raft_consensus.cc:504] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.517450 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:11.534017 16448 leader_election.cc:341] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:11.536266 16450 leader_election.cc:302] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:11.537396 16855 raft_consensus.cc:2781] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:11.537845 16855 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:11.538267 16855 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:11.518173 16907 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:11.549113 17121 leader_election.cc:288] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:11.550107 17025 raft_consensus.cc:482] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:11.558014 17025 raft_consensus.cc:504] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.560330 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:51:11.561954 16448 leader_election.cc:341] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:11.562567 17025 leader_election.cc:288] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:11.564651 16818 consensus_queue.cc:260] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.566946 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:11.567970 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:51:11.571559 16695 leader_election.cc:341] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:11.572029 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:11.572532 16505 raft_consensus.cc:3037] T 3e58f8d2231b4e2d900ec099ae700d09 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:11.572765 16748 raft_consensus.cc:3037] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:11.575505 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:11.564443 17116 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:11.583052 16748 raft_consensus.cc:2454] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
W20260810 18:51:11.584875 16573 leader_election.cc:341] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:11.585635 16450 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:11.589118 16907 raft_consensus.cc:2781] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:11.590003 16907 raft_consensus.cc:686] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:11.591517 16907 consensus_queue.cc:237] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.595804 17177 consensus_queue.cc:1035] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:11.599700 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:11.601868 16575 leader_election.cc:341] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:11.602698 16575 leader_election.cc:302] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:11.603296 16818 ts_tablet_manager.cc:1428] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.100s user 0.019s sys 0.011s
I20260810 18:51:11.604831 16818 tablet_bootstrap.cc:492] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:11.607332 17033 raft_consensus.cc:2726] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 18:51:11.610335 16701 leader_election.cc:341] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:11.611114 16701 leader_election.cc:302] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:11.612375 17025 raft_consensus.cc:2726] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:11.618701 16817 tablet_bootstrap.cc:492] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:11.671492 16818 tablet_bootstrap.cc:654] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:11.727510 16817 ts_tablet_manager.cc:1397] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.246s user 0.037s sys 0.045s
I20260810 18:51:11.711567 17177 raft_consensus.cc:482] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:51:11.717334 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 81107us
I20260810 18:51:11.613305 16855 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.746569 16907 raft_consensus.cc:482] T 3e58f8d2231b4e2d900ec099ae700d09 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:11.755309 16746 raft_consensus.cc:2454] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
W20260810 18:51:11.748414 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 60103.1us
I20260810 18:51:11.755774 16907 raft_consensus.cc:504] T 3e58f8d2231b4e2d900ec099ae700d09 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.759672 16817 raft_consensus.cc:348] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.775869 16817 raft_consensus.cc:374] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:11.776193 16817 raft_consensus.cc:729] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:11.778067 16817 consensus_queue.cc:260] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.781879 16575 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:11.783538 17033 raft_consensus.cc:2781] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:11.784252 17033 raft_consensus.cc:686] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:11.753326 17116 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:11.789932 16817 ts_tablet_manager.cc:1428] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.062s user 0.006s sys 0.009s
I20260810 18:51:11.794240 17116 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.796972 17196 raft_consensus.cc:482] T c3b33c2c321842498975ee1086991616 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:11.797390 16855 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:11.797711 17116 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:11.797582 17196 raft_consensus.cc:504] T c3b33c2c321842498975ee1086991616 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.800380 17033 consensus_queue.cc:237] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.784269 17121 consensus_queue.cc:1035] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20260810 18:51:11.801908 17196 leader_election.cc:288] T c3b33c2c321842498975ee1086991616 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:11.750438 17177 raft_consensus.cc:504] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.809862 16907 leader_election.cc:288] T 3e58f8d2231b4e2d900ec099ae700d09 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:11.817481 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:11.818437 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:11.819677 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:11.820119 16626 raft_consensus.cc:2379] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 1: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:51:11.820647 16636 raft_consensus.cc:2454] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 0.
I20260810 18:51:11.819253 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:11.828477 16448 leader_election.cc:341] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:11.847529 16958 consensus_queue.cc:1035] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:11.838428 17177 leader_election.cc:288] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:11.848296 17048 consensus_queue.cc:1035] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
W20260810 18:51:11.850340 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 67169.3us
I20260810 18:51:11.860646 16448 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:11.861436 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:11.862221 17200 raft_consensus.cc:2781] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:11.862520 16507 raft_consensus.cc:2379] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 1: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:51:11.862689 17200 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:11.863161 17200 raft_consensus.cc:3037] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:11.869344 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:11.870167 16748 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:11.875247 16817 tablet_bootstrap.cc:492] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:11.877651 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:11.870213 16342 catalog_manager.cc:5495] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 0 to 1, leader changed from <none> to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:11.889286 16814 tablet_bootstrap.cc:654] T b4248aab7d8b4ed0b7c298cad63430f8 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:11.898818 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:11.920524 16508 raft_consensus.cc:3037] T c3b33c2c321842498975ee1086991616 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:11.918895 16631 raft_consensus.cc:3037] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:11.926940 16575 leader_election.cc:302] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:11.931636 16818 tablet_bootstrap.cc:492] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:11.932360 16818 ts_tablet_manager.cc:1397] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.328s user 0.017s sys 0.012s
I20260810 18:51:11.943308 17033 raft_consensus.cc:2726] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:11.930260 17199 consensus_queue.cc:1035] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260810 18:51:11.959170 17048 consensus_queue.cc:1035] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:11.964919 16748 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
W20260810 18:51:11.968969 16450 leader_election.cc:341] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:11.970932 17048 raft_consensus.cc:482] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:11.972630 16958 consensus_queue.cc:1035] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:11.981381 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:11.975322 17048 raft_consensus.cc:504] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:11.983441 17048 leader_election.cc:288] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:11.983237 16338 catalog_manager.cc:5495] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:11.990813 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:11.995642 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:11.999634 16450 leader_election.cc:302] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:12.004575 17195 raft_consensus.cc:2781] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:12.005264 17195 raft_consensus.cc:686] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:12.005841 16818 raft_consensus.cc:348] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.006748 16818 raft_consensus.cc:374] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:12.007191 16818 raft_consensus.cc:729] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:12.006517 17195 consensus_queue.cc:237] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.008015 16818 consensus_queue.cc:260] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.010603 16818 ts_tablet_manager.cc:1428] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.078s user 0.006s sys 0.001s
I20260810 18:51:12.011634 16818 tablet_bootstrap.cc:492] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:12.019014 16450 leader_election.cc:302] T c3b33c2c321842498975ee1086991616 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:12.020459 17201 raft_consensus.cc:2726] T c3b33c2c321842498975ee1086991616 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:12.020840 16450 leader_election.cc:302] T 3e58f8d2231b4e2d900ec099ae700d09 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:12.021692 17201 raft_consensus.cc:2726] T 3e58f8d2231b4e2d900ec099ae700d09 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:12.012152 17200 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:12.014576 16701 leader_election.cc:341] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:12.031481 16958 consensus_queue.cc:1035] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:51:12.031939 17200 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:12.033931 16814 tablet_bootstrap.cc:492] T b4248aab7d8b4ed0b7c298cad63430f8 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:12.034919 16814 ts_tablet_manager.cc:1397] T b4248aab7d8b4ed0b7c298cad63430f8 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.528s user 0.004s sys 0.027s
I20260810 18:51:12.031996 16817 tablet_bootstrap.cc:654] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:12.048033 16814 raft_consensus.cc:348] T b4248aab7d8b4ed0b7c298cad63430f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.048740 16814 raft_consensus.cc:374] T b4248aab7d8b4ed0b7c298cad63430f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:12.049067 16814 raft_consensus.cc:729] T b4248aab7d8b4ed0b7c298cad63430f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:12.049811 16814 consensus_queue.cc:260] T b4248aab7d8b4ed0b7c298cad63430f8 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.052649 16814 ts_tablet_manager.cc:1428] T b4248aab7d8b4ed0b7c298cad63430f8 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.014s user 0.005s sys 0.002s
I20260810 18:51:12.053720 16814 tablet_bootstrap.cc:492] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:12.059718 16818 tablet_bootstrap.cc:654] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:12.066015 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:12.066917 16636 raft_consensus.cc:3037] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:12.069895 16814 tablet_bootstrap.cc:654] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:12.078188 16817 tablet_bootstrap.cc:492] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:12.078677 16817 ts_tablet_manager.cc:1397] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.205s user 0.015s sys 0.002s
I20260810 18:51:12.083841 17033 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:12.101590 17033 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:12.098467 16695 consensus_peers.cc:487] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be -> Peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Couldn't send request to peer d304ac47fdf846de81337057a4ef7bea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:12.106534 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:12.113384 16628 raft_consensus.cc:3037] T b4248aab7d8b4ed0b7c298cad63430f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:12.114328 16636 raft_consensus.cc:2454] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:12.128772 16448 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:12.131163 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:12.137670 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:12.140908 16907 raft_consensus.cc:2781] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:12.144357 16907 raft_consensus.cc:686] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:12.157404 16817 raft_consensus.cc:348] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.185981 16817 raft_consensus.cc:374] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:12.187049 16817 raft_consensus.cc:729] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:12.187441 16818 tablet_bootstrap.cc:492] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:12.187891 16818 ts_tablet_manager.cc:1397] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.176s user 0.013s sys 0.011s
W20260810 18:51:12.158090 16695 leader_election.cc:341] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:12.163266 17033 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:12.182137 17201 raft_consensus.cc:482] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:12.187824 16817 consensus_queue.cc:260] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:12.188172 16450 leader_election.cc:341] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:12.204711 17201 raft_consensus.cc:504] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.208351 17201 leader_election.cc:288] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:12.214128 16817 ts_tablet_manager.cc:1428] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.135s user 0.010s sys 0.000s
I20260810 18:51:12.215353 16817 tablet_bootstrap.cc:492] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:12.216150 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:12.201969 16695 leader_election.cc:302] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:12.214146 16907 consensus_queue.cc:237] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:12.225459 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 68768.2us
I20260810 18:51:12.226938 17143 raft_consensus.cc:2726] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:12.227260 17200 consensus_queue.cc:1035] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:51:12.286482 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:12.297219 16814 tablet_bootstrap.cc:492] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:12.296912 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:12.297788 16814 ts_tablet_manager.cc:1397] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.244s user 0.041s sys 0.067s
I20260810 18:51:12.298158 16746 raft_consensus.cc:2379] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 1: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:12.300894 16345 catalog_manager.cc:5495] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 0 to 1, leader changed from <none> to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:12.308372 16814 raft_consensus.cc:348] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.309101 16814 raft_consensus.cc:374] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:12.309506 16814 raft_consensus.cc:729] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:12.310448 16814 consensus_queue.cc:260] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.312817 16814 ts_tablet_manager.cc:1428] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.015s user 0.005s sys 0.006s
I20260810 18:51:12.314256 16814 tablet_bootstrap.cc:492] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:12.321903 16817 tablet_bootstrap.cc:654] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:12.333679 16814 tablet_bootstrap.cc:654] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:12.337858 16818 raft_consensus.cc:348] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.339172 16818 raft_consensus.cc:374] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:12.339498 16818 raft_consensus.cc:729] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:12.341516 16818 consensus_queue.cc:260] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:12.340804 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 81382.6us
I20260810 18:51:12.356604 16818 ts_tablet_manager.cc:1428] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.168s user 0.010s sys 0.007s
I20260810 18:51:12.357534 16818 tablet_bootstrap.cc:492] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:12.363197 16575 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:12.365808 16448 leader_election.cc:302] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:12.366266 16912 raft_consensus.cc:2726] T b4248aab7d8b4ed0b7c298cad63430f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:12.367544 17189 consensus_queue.cc:1035] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:51:12.369182 17189 raft_consensus.cc:2726] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:12.380945 17121 consensus_queue.cc:1035] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:12.402272 16506 raft_consensus.cc:3037] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
W20260810 18:51:12.416740 16415 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260810 18:51:12.432269 16817 tablet_bootstrap.cc:492] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:12.432850 16817 ts_tablet_manager.cc:1397] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.218s user 0.017s sys 0.009s
W20260810 18:51:12.436146 16573 consensus_peers.cc:487] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb -> Peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Couldn't send request to peer d304ac47fdf846de81337057a4ef7bea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:12.438303 16817 raft_consensus.cc:348] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.439169 16817 raft_consensus.cc:374] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:12.439467 16817 raft_consensus.cc:729] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:12.446005 16814 tablet_bootstrap.cc:492] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:12.446614 16814 ts_tablet_manager.cc:1397] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.133s user 0.022s sys 0.030s
I20260810 18:51:12.440114 16817 consensus_queue.cc:260] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.450644 16817 ts_tablet_manager.cc:1428] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260810 18:51:12.451625 16817 tablet_bootstrap.cc:492] T ab9362a1df854c4fad0204ee2f9f78ad P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:12.457777 16814 raft_consensus.cc:348] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.458684 16814 raft_consensus.cc:374] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:12.458968 16814 raft_consensus.cc:729] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:12.459770 16814 consensus_queue.cc:260] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.459782 16338 catalog_manager.cc:5495] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 0 to 1, leader changed from <none> to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:12.465409 16814 ts_tablet_manager.cc:1428] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.018s user 0.010s sys 0.002s
I20260810 18:51:12.466955 16814 tablet_bootstrap.cc:492] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:12.472632 16817 tablet_bootstrap.cc:654] T ab9362a1df854c4fad0204ee2f9f78ad P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:12.477350 16818 tablet_bootstrap.cc:654] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:12.484156 16814 tablet_bootstrap.cc:654] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:12.498358 16818 tablet_bootstrap.cc:492] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:12.499215 16818 ts_tablet_manager.cc:1397] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.142s user 0.016s sys 0.000s
I20260810 18:51:12.503175 16818 raft_consensus.cc:348] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.504261 16818 raft_consensus.cc:374] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:12.504657 16818 raft_consensus.cc:729] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:12.505545 16818 consensus_queue.cc:260] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.508755 16818 ts_tablet_manager.cc:1428] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260810 18:51:12.510032 16818 tablet_bootstrap.cc:492] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:12.510798 16817 tablet_bootstrap.cc:492] T ab9362a1df854c4fad0204ee2f9f78ad P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:12.511293 16817 ts_tablet_manager.cc:1397] T ab9362a1df854c4fad0204ee2f9f78ad P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.060s user 0.010s sys 0.012s
I20260810 18:51:12.515110 16817 raft_consensus.cc:348] T ab9362a1df854c4fad0204ee2f9f78ad P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.516091 16817 raft_consensus.cc:374] T ab9362a1df854c4fad0204ee2f9f78ad P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:12.516402 16817 raft_consensus.cc:729] T ab9362a1df854c4fad0204ee2f9f78ad P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:12.517714 16817 consensus_queue.cc:260] T ab9362a1df854c4fad0204ee2f9f78ad P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.519985 16818 tablet_bootstrap.cc:654] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:12.520570 16817 ts_tablet_manager.cc:1428] T ab9362a1df854c4fad0204ee2f9f78ad P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s
I20260810 18:51:12.522029 16817 tablet_bootstrap.cc:492] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:12.537546 16814 tablet_bootstrap.cc:492] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:12.538048 16814 ts_tablet_manager.cc:1397] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.071s user 0.027s sys 0.028s
I20260810 18:51:12.538880 16817 tablet_bootstrap.cc:654] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:12.541666 16814 raft_consensus.cc:348] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.541929 16818 tablet_bootstrap.cc:492] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:12.542284 16814 raft_consensus.cc:374] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:12.542583 16814 raft_consensus.cc:729] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:12.542740 16818 ts_tablet_manager.cc:1397] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.033s user 0.012s sys 0.012s
I20260810 18:51:12.543213 16814 consensus_queue.cc:260] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.545801 16814 ts_tablet_manager.cc:1428] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.007s user 0.005s sys 0.004s
I20260810 18:51:12.547508 16818 raft_consensus.cc:348] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.548431 16818 raft_consensus.cc:374] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:12.548744 16818 raft_consensus.cc:729] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:12.549729 16818 consensus_queue.cc:260] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.552815 16818 ts_tablet_manager.cc:1428] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20260810 18:51:12.553455 16631 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:12.553998 17245 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:12.554896 17245 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.561743 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:12.566202 17245 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:12.567535 16818 tablet_bootstrap.cc:492] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:12.575297 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:12.588682 16450 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:12.590211 17245 raft_consensus.cc:2726] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:12.597082 17195 consensus_queue.cc:1035] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:12.598901 16818 tablet_bootstrap.cc:654] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:12.603360 16814 tablet_bootstrap.cc:492] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:12.609552 17189 consensus_queue.cc:1035] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:12.621647 17189 consensus_queue.cc:1035] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:12.644345 17245 raft_consensus.cc:482] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:12.645067 17245 raft_consensus.cc:504] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:12.650082 16450 consensus_peers.cc:487] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea -> Peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Couldn't send request to peer 980f343dd1fd4e049143f8b9d9f2f1be. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:12.664672 17048 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:12.665618 17048 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.671334 17048 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:12.671017 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:12.677802 17121 raft_consensus.cc:482] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:12.678583 17121 raft_consensus.cc:504] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:12.739501 16695 leader_election.cc:341] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:12.741815 17048 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:12.742513 17048 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.744940 17189 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:12.745805 17048 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:12.745743 17189 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.755893 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:12.759019 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:12.760105 16633 raft_consensus.cc:2454] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 0.
I20260810 18:51:12.763880 16701 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:12.765269 16958 raft_consensus.cc:2781] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:12.766013 16958 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:12.766741 16958 raft_consensus.cc:3037] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:12.772073 17121 leader_election.cc:288] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:12.773533 17189 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:12.775377 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:12.777310 16817 tablet_bootstrap.cc:492] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:12.777273 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:51:12.777554 16450 leader_election.cc:341] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:12.777992 16817 ts_tablet_manager.cc:1397] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.256s user 0.048s sys 0.036s
I20260810 18:51:12.778136 16507 raft_consensus.cc:2454] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 0.
W20260810 18:51:12.794248 16695 leader_election.cc:341] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:12.794883 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:12.795336 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:12.796144 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:12.797786 16701 leader_election.cc:341] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:12.798311 16701 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:12.798628 16631 raft_consensus.cc:2454] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 0.
W20260810 18:51:12.800323 16448 leader_election.cc:341] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:12.800951 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:12.807650 16746 raft_consensus.cc:2454] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 0.
I20260810 18:51:12.809345 16448 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:12.811489 17189 raft_consensus.cc:2726] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:12.812664 17245 leader_election.cc:288] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:12.814085 16573 leader_election.cc:302] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:12.817178 16889 raft_consensus.cc:2726] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:12.818411 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:12.819664 16750 raft_consensus.cc:2454] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 0.
I20260810 18:51:12.778177 16958 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.821142 17143 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:12.832449 17121 raft_consensus.cc:2781] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:12.833528 16817 raft_consensus.cc:348] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.834352 16817 raft_consensus.cc:374] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:12.834610 16817 raft_consensus.cc:729] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:12.835408 16450 leader_election.cc:302] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:12.833078 17121 raft_consensus.cc:482] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:12.838157 17121 raft_consensus.cc:3037] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:12.838335 16958 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:12.831161 17143 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.845932 17143 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:12.836846 16817 consensus_queue.cc:260] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.855679 16818 tablet_bootstrap.cc:492] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:12.856235 16818 ts_tablet_manager.cc:1397] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.289s user 0.014s sys 0.004s
I20260810 18:51:12.859920 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:12.857149 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:12.866096 16817 ts_tablet_manager.cc:1428] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.088s user 0.008s sys 0.003s
I20260810 18:51:12.855511 16907 raft_consensus.cc:2781] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:12.869158 16907 raft_consensus.cc:482] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:12.869721 16907 raft_consensus.cc:3037] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:12.876106 16818 raft_consensus.cc:348] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:12.879387 16695 leader_election.cc:341] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
W20260810 18:51:12.882074 16695 leader_election.cc:341] T ab9362a1df854c4fad0204ee2f9f78ad P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:12.887434 16817 tablet_bootstrap.cc:492] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:12.896049 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:12.916421 16628 raft_consensus.cc:3037] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:12.895277 17121 raft_consensus.cc:504] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.898221 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:12.920998 17121 leader_election.cc:288] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:12.924280 16628 raft_consensus.cc:2454] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:12.926913 16907 raft_consensus.cc:504] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.929104 16907 leader_election.cc:288] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:12.929915 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:12.930450 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:12.879719 16818 raft_consensus.cc:374] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:12.931257 16818 raft_consensus.cc:729] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:12.930569 16746 raft_consensus.cc:3037] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:12.931384 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:12.932050 16818 consensus_queue.cc:260] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.900988 16814 tablet_bootstrap.cc:654] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:12.934835 16818 ts_tablet_manager.cc:1428] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.078s user 0.009s sys 0.000s
I20260810 18:51:12.935269 16636 raft_consensus.cc:2379] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 1: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
W20260810 18:51:12.939500 16701 leader_election.cc:341] T ab9362a1df854c4fad0204ee2f9f78ad P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:12.940243 16701 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:12.935827 16818 tablet_bootstrap.cc:492] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:12.943646 17048 raft_consensus.cc:2726] T ab9362a1df854c4fad0204ee2f9f78ad P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:12.948148 16504 raft_consensus.cc:3037] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:12.950049 16701 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:12.952021 17048 raft_consensus.cc:2781] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:12.954066 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:12.955433 16505 raft_consensus.cc:2379] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 1: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:12.956413 16817 tablet_bootstrap.cc:654] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:12.962927 16818 tablet_bootstrap.cc:654] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:12.964331 17048 raft_consensus.cc:686] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:12.965947 17048 consensus_queue.cc:237] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:12.970271 16450 leader_election.cc:302] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:12.971254 16907 raft_consensus.cc:2726] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 18:51:12.984019 16746 raft_consensus.cc:2454] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:12.987934 16818 tablet_bootstrap.cc:492] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:12.988390 16818 ts_tablet_manager.cc:1397] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.053s user 0.008s sys 0.017s
I20260810 18:51:12.993553 16814 tablet_bootstrap.cc:492] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:12.994117 16814 ts_tablet_manager.cc:1397] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.391s user 0.007s sys 0.035s
I20260810 18:51:12.994616 16575 leader_election.cc:302] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:12.999886 16818 raft_consensus.cc:348] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.000813 16818 raft_consensus.cc:374] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:13.001096 16818 raft_consensus.cc:729] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:13.002058 16818 consensus_queue.cc:260] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.001701 17033 raft_consensus.cc:2781] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:13.006069 17033 raft_consensus.cc:686] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:13.007167 16818 ts_tablet_manager.cc:1428] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.019s user 0.009s sys 0.000s
I20260810 18:51:13.008271 16818 tablet_bootstrap.cc:492] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:13.007688 17033 consensus_queue.cc:237] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.011001 16342 catalog_manager.cc:5495] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 0 to 1, leader changed from <none> to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:51:13.016088 16814 raft_consensus.cc:348] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.016826 16814 raft_consensus.cc:374] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:13.017086 16814 raft_consensus.cc:729] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:13.017233 16817 tablet_bootstrap.cc:492] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:13.017661 16817 ts_tablet_manager.cc:1397] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.131s user 0.016s sys 0.010s
I20260810 18:51:13.017699 16814 consensus_queue.cc:260] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.021483 16814 ts_tablet_manager.cc:1428] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.027s user 0.006s sys 0.001s
I20260810 18:51:13.026208 16814 tablet_bootstrap.cc:492] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:13.028992 16817 raft_consensus.cc:348] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.029604 16817 raft_consensus.cc:374] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:13.029942 16817 raft_consensus.cc:729] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:13.030584 16817 consensus_queue.cc:260] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.034394 16817 ts_tablet_manager.cc:1428] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.016s user 0.001s sys 0.006s
I20260810 18:51:13.035394 16817 tablet_bootstrap.cc:492] T c758b1e1139941a9a8c4e703cc058061 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:13.042773 16818 tablet_bootstrap.cc:654] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:13.048319 16345 catalog_manager.cc:5495] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:13.050843 17195 raft_consensus.cc:482] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:13.051496 17195 raft_consensus.cc:504] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.054762 17195 leader_election.cc:288] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:13.055518 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:13.055711 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:13.056373 16817 tablet_bootstrap.cc:654] T c758b1e1139941a9a8c4e703cc058061 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
W20260810 18:51:13.056707 16450 leader_election.cc:341] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:13.058394 16814 tablet_bootstrap.cc:654] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
W20260810 18:51:13.057451 16448 leader_election.cc:341] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:13.059242 16448 leader_election.cc:302] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:13.060211 16907 raft_consensus.cc:2726] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:13.070124 16818 tablet_bootstrap.cc:492] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:13.070581 16818 ts_tablet_manager.cc:1397] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.063s user 0.015s sys 0.001s
I20260810 18:51:13.072892 16818 raft_consensus.cc:348] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.073601 16818 raft_consensus.cc:374] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:13.073885 16818 raft_consensus.cc:729] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:13.075098 16817 tablet_bootstrap.cc:492] T c758b1e1139941a9a8c4e703cc058061 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:13.074476 16818 consensus_queue.cc:260] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.075687 16817 ts_tablet_manager.cc:1397] T c758b1e1139941a9a8c4e703cc058061 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.040s user 0.012s sys 0.000s
I20260810 18:51:13.077363 16818 ts_tablet_manager.cc:1428] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20260810 18:51:13.078616 16818 tablet_bootstrap.cc:492] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:13.079263 16817 raft_consensus.cc:348] T c758b1e1139941a9a8c4e703cc058061 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.079946 16817 raft_consensus.cc:374] T c758b1e1139941a9a8c4e703cc058061 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:13.080200 16817 raft_consensus.cc:729] T c758b1e1139941a9a8c4e703cc058061 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:13.080807 16817 consensus_queue.cc:260] T c758b1e1139941a9a8c4e703cc058061 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.083778 16817 ts_tablet_manager.cc:1428] T c758b1e1139941a9a8c4e703cc058061 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260810 18:51:13.084649 16817 tablet_bootstrap.cc:492] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:13.084479 17048 raft_consensus.cc:482] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:13.085165 17048 raft_consensus.cc:504] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.088354 17048 leader_election.cc:288] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:13.090170 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:13.090970 16636 raft_consensus.cc:2454] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 0.
I20260810 18:51:13.092448 16701 leader_election.cc:302] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:13.093619 17048 raft_consensus.cc:2781] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:13.094403 17048 raft_consensus.cc:482] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:13.095050 17048 raft_consensus.cc:3037] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:13.104146 16814 tablet_bootstrap.cc:492] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:13.114769 17048 raft_consensus.cc:504] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.145373 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:13.146410 16636 raft_consensus.cc:3037] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:13.148028 16814 ts_tablet_manager.cc:1397] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.122s user 0.013s sys 0.021s
I20260810 18:51:13.145562 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:13.149710 17048 leader_election.cc:288] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:13.151467 16818 tablet_bootstrap.cc:654] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:13.152734 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:13.154830 16814 raft_consensus.cc:348] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:13.160035 16695 leader_election.cc:341] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:13.160275 16636 raft_consensus.cc:2454] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:13.155596 16814 raft_consensus.cc:374] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:13.165360 16701 leader_election.cc:302] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:13.166641 17048 raft_consensus.cc:2781] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:13.167352 17048 raft_consensus.cc:686] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:13.168437 17048 consensus_queue.cc:237] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.171264 17025 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:13.171952 17025 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.164652 16814 raft_consensus.cc:729] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:13.177517 16818 tablet_bootstrap.cc:492] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:13.178122 16818 ts_tablet_manager.cc:1397] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.100s user 0.030s sys 0.023s
I20260810 18:51:13.182981 16814 consensus_queue.cc:260] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:13.180974 16695 leader_election.cc:341] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:13.186851 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:13.187098 16818 raft_consensus.cc:348] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.187927 16818 raft_consensus.cc:374] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:13.188218 16818 raft_consensus.cc:729] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:13.188803 16818 consensus_queue.cc:260] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.191347 16818 ts_tablet_manager.cc:1428] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260810 18:51:13.192359 16818 tablet_bootstrap.cc:492] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:13.207217 16345 catalog_manager.cc:5495] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 0 to 1, leader changed from <none> to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:51:13.209529 16746 raft_consensus.cc:3037] T c758b1e1139941a9a8c4e703cc058061 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:13.209753 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:13.210301 16814 ts_tablet_manager.cc:1428] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.062s user 0.010s sys 0.000s
I20260810 18:51:13.211252 16814 tablet_bootstrap.cc:492] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:13.211740 16701 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:13.216480 16818 tablet_bootstrap.cc:654] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:13.203730 17025 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:13.219987 17195 raft_consensus.cc:482] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:13.220561 17195 raft_consensus.cc:504] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.225354 17195 leader_election.cc:288] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:13.227360 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:13.231246 16817 tablet_bootstrap.cc:654] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:13.231158 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:13.234977 16814 tablet_bootstrap.cc:654] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:13.235589 17025 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:13.236292 16631 raft_consensus.cc:2379] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 1: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:51:13.250936 16448 leader_election.cc:302] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:13.240850 17025 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.252972 17195 raft_consensus.cc:3037] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:13.258490 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:13.259457 16818 tablet_bootstrap.cc:492] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:13.260267 16818 ts_tablet_manager.cc:1397] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.068s user 0.015s sys 0.024s
I20260810 18:51:13.263356 16817 tablet_bootstrap.cc:492] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:13.263993 16817 ts_tablet_manager.cc:1397] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.180s user 0.010s sys 0.009s
I20260810 18:51:13.265153 17199 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:13.265869 17199 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.269321 16907 raft_consensus.cc:482] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:13.270056 16907 raft_consensus.cc:504] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:13.272446 16695 leader_election.cc:341] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:13.270563 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:13.272616 17196 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:13.280663 17196 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:13.282833 16573 leader_election.cc:341] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:13.284003 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:13.287581 16818 raft_consensus.cc:348] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.288544 16818 raft_consensus.cc:374] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:13.288863 16818 raft_consensus.cc:729] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
W20260810 18:51:13.288851 16448 leader_election.cc:341] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:13.289973 17199 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:13.290838 16817 raft_consensus.cc:348] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.289708 16818 consensus_queue.cc:260] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.293526 17196 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:13.294009 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:13.294859 16752 raft_consensus.cc:2454] T ab9362a1df854c4fad0204ee2f9f78ad P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 0.
I20260810 18:51:13.295221 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:13.295800 16814 tablet_bootstrap.cc:492] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:13.295903 16748 raft_consensus.cc:2454] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 0.
I20260810 18:51:13.296347 16814 ts_tablet_manager.cc:1397] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.085s user 0.025s sys 0.023s
I20260810 18:51:13.296213 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:13.297179 16575 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:13.298532 16450 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:13.299201 17199 raft_consensus.cc:2781] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:13.299566 17199 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:13.292306 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:13.299974 17199 raft_consensus.cc:3037] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
W20260810 18:51:13.300275 16701 leader_election.cc:341] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:13.292572 17025 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:13.300695 16701 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:13.300558 16907 leader_election.cc:288] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:13.301781 16958 raft_consensus.cc:2726] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:13.305081 16818 ts_tablet_manager.cc:1428] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.044s user 0.008s sys 0.000s
I20260810 18:51:13.305019 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:13.306881 17245 raft_consensus.cc:2781] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:13.307307 17245 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:13.307636 16818 tablet_bootstrap.cc:492] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:13.293040 16817 raft_consensus.cc:374] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:13.308171 16817 raft_consensus.cc:729] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
W20260810 18:51:13.308238 16450 leader_election.cc:341] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:13.307637 17245 raft_consensus.cc:3037] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:13.311645 17048 raft_consensus.cc:2726] T c758b1e1139941a9a8c4e703cc058061 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:13.308646 16450 leader_election.cc:302] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:13.313082 16817 consensus_queue.cc:260] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.313649 17196 raft_consensus.cc:2726] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:13.318871 16814 raft_consensus.cc:348] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.319525 16814 raft_consensus.cc:374] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:13.319757 16814 raft_consensus.cc:729] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:13.320992 16817 ts_tablet_manager.cc:1428] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.057s user 0.009s sys 0.000s
I20260810 18:51:13.320770 16814 consensus_queue.cc:260] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.322028 16817 tablet_bootstrap.cc:492] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:13.316246 17195 raft_consensus.cc:2726] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:13.323828 17033 raft_consensus.cc:482] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:13.324401 17033 raft_consensus.cc:504] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.337792 17033 leader_election.cc:288] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:13.339788 16818 tablet_bootstrap.cc:654] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:13.339568 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:13.339871 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:13.340917 16505 raft_consensus.cc:2454] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 0.
W20260810 18:51:13.340893 16575 leader_election.cc:341] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:13.342281 16573 leader_election.cc:302] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:13.344204 17033 raft_consensus.cc:2781] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:13.344558 17033 raft_consensus.cc:482] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:13.344887 17033 raft_consensus.cc:3037] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:13.338210 16631 raft_consensus.cc:2454] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 0.
I20260810 18:51:13.349113 17199 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.341001 17245 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.354125 17245 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:13.355404 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:13.355572 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:13.356050 16631 raft_consensus.cc:3037] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:13.356588 16746 raft_consensus.cc:3037] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:13.357632 16814 ts_tablet_manager.cc:1428] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.061s user 0.008s sys 0.000s
I20260810 18:51:13.358011 16817 tablet_bootstrap.cc:654] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:13.358616 16814 tablet_bootstrap.cc:492] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:13.361598 17199 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:13.361784 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:13.362867 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
W20260810 18:51:13.363099 16573 leader_election.cc:341] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:13.363490 16750 raft_consensus.cc:3037] T ab9362a1df854c4fad0204ee2f9f78ad P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:13.373729 16750 raft_consensus.cc:2454] T ab9362a1df854c4fad0204ee2f9f78ad P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:13.375281 16575 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:13.376642 17199 raft_consensus.cc:2781] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:13.377453 17199 raft_consensus.cc:686] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:13.378840 17199 consensus_queue.cc:237] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.400309 16631 raft_consensus.cc:2454] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:13.414402 16746 raft_consensus.cc:2454] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:13.417161 17048 consensus_queue.cc:1035] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260810 18:51:13.435606 16817 tablet_bootstrap.cc:492] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:13.436115 16817 ts_tablet_manager.cc:1397] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.114s user 0.028s sys 0.027s
I20260810 18:51:13.441263 16448 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:13.442523 17195 raft_consensus.cc:2781] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:13.443143 17195 raft_consensus.cc:686] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:13.444145 17195 consensus_queue.cc:237] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.416759 17033 raft_consensus.cc:504] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.455881 17033 leader_election.cc:288] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:13.456422 17196 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:13.457024 17196 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.458699 16814 tablet_bootstrap.cc:654] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:13.460920 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:13.463537 16342 catalog_manager.cc:5495] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:13.461298 17196 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:13.469304 16818 tablet_bootstrap.cc:492] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:13.469858 16818 ts_tablet_manager.cc:1397] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.162s user 0.010s sys 0.033s
I20260810 18:51:13.473068 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:13.473335 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:13.475435 16450 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:13.477638 17245 raft_consensus.cc:2726] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:13.477468 16818 raft_consensus.cc:348] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.478250 16818 raft_consensus.cc:374] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:13.478544 16818 raft_consensus.cc:729] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:13.479468 16818 consensus_queue.cc:260] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.481457 16818 ts_tablet_manager.cc:1428] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
W20260810 18:51:13.482208 16575 leader_election.cc:341] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:13.485082 16818 tablet_bootstrap.cc:492] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:13.484069 16817 raft_consensus.cc:348] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.486963 16817 raft_consensus.cc:374] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:13.487286 16817 raft_consensus.cc:729] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:13.487847 16817 consensus_queue.cc:260] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.491947 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:13.492818 16504 raft_consensus.cc:3037] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:13.494354 16817 ts_tablet_manager.cc:1428] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.058s user 0.008s sys 0.000s
I20260810 18:51:13.495551 16817 tablet_bootstrap.cc:492] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:13.494864 16342 catalog_manager.cc:5495] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 0 to 1, leader changed from <none> to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:13.515379 17196 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:13.515893 17196 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.530350 16818 tablet_bootstrap.cc:654] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:13.521549 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:13.532943 16448 leader_election.cc:341] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:13.546356 17310 consensus_queue.cc:1035] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:13.548115 16505 raft_consensus.cc:3037] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:13.555425 17310 consensus_queue.cc:1035] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:13.558360 17196 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:13.565143 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:13.565986 16750 raft_consensus.cc:2454] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 0.
I20260810 18:51:13.567468 16450 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:13.569417 16504 raft_consensus.cc:2454] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:13.569643 17196 raft_consensus.cc:2781] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:13.570314 17196 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:13.570922 17196 raft_consensus.cc:3037] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:13.570863 16573 leader_election.cc:302] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:13.580601 17048 consensus_queue.cc:1035] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:13.581687 16814 tablet_bootstrap.cc:492] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:13.582201 16814 ts_tablet_manager.cc:1397] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.224s user 0.013s sys 0.004s
I20260810 18:51:13.584408 17313 raft_consensus.cc:2781] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:13.587504 17313 raft_consensus.cc:686] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:13.588816 16818 tablet_bootstrap.cc:492] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:13.589334 16818 ts_tablet_manager.cc:1397] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.104s user 0.013s sys 0.004s
I20260810 18:51:13.591948 16818 raft_consensus.cc:348] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.592689 16818 raft_consensus.cc:374] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:13.592985 16818 raft_consensus.cc:729] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:13.585151 16814 raft_consensus.cc:348] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.593701 16814 raft_consensus.cc:374] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:13.594038 16814 raft_consensus.cc:729] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:13.596688 16814 consensus_queue.cc:260] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.558355 16817 tablet_bootstrap.cc:654] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:13.598415 16814 ts_tablet_manager.cc:1428] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260810 18:51:13.603286 16814 tablet_bootstrap.cc:492] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:13.604954 16818 consensus_queue.cc:260] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.590237 17313 consensus_queue.cc:237] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.623176 16818 ts_tablet_manager.cc:1428] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.034s user 0.007s sys 0.000s
I20260810 18:51:13.624145 16818 tablet_bootstrap.cc:492] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:13.645920 17121 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:13.648306 17121 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.654853 17196 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.662315 17121 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:13.685283 16817 tablet_bootstrap.cc:492] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:13.685777 16817 ts_tablet_manager.cc:1397] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.190s user 0.023s sys 0.020s
I20260810 18:51:13.676143 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:13.689287 17048 consensus_queue.cc:1035] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:51:13.689426 16748 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 0.
I20260810 18:51:13.694388 16814 tablet_bootstrap.cc:654] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:13.695027 16575 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:13.696283 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:13.697093 16505 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 0.
W20260810 18:51:13.687527 16415 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260810 18:51:13.702929 17196 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:13.702790 16342 catalog_manager.cc:5495] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:13.710961 16817 raft_consensus.cc:348] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.702838 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:13.718658 16817 raft_consensus.cc:374] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:13.710637 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:13.728322 17332 raft_consensus.cc:2781] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:13.733695 16817 raft_consensus.cc:729] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:13.737915 16817 consensus_queue.cc:260] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.740821 16817 ts_tablet_manager.cc:1428] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.055s user 0.009s sys 0.001s
I20260810 18:51:13.745178 16818 tablet_bootstrap.cc:654] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:13.753381 16958 consensus_queue.cc:1035] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:51:13.742740 16748 raft_consensus.cc:3037] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:13.761058 16817 tablet_bootstrap.cc:492] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:13.759382 17332 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:13.769248 17332 raft_consensus.cc:3037] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
W20260810 18:51:13.770841 16448 leader_election.cc:341] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:13.802982 16817 tablet_bootstrap.cc:654] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:13.815431 16958 raft_consensus.cc:482] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:13.816525 16814 tablet_bootstrap.cc:492] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:13.816368 16958 raft_consensus.cc:504] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.817319 16814 ts_tablet_manager.cc:1397] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.214s user 0.013s sys 0.048s
I20260810 18:51:13.823057 16818 tablet_bootstrap.cc:492] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:13.823660 16818 ts_tablet_manager.cc:1397] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.200s user 0.016s sys 0.007s
I20260810 18:51:13.881019 16907 raft_consensus.cc:482] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:13.882668 16907 raft_consensus.cc:504] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.883863 16818 raft_consensus.cc:348] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.885044 16818 raft_consensus.cc:374] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:13.885439 16818 raft_consensus.cc:729] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:13.886305 16818 consensus_queue.cc:260] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.891460 16814 raft_consensus.cc:348] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.892640 16814 raft_consensus.cc:374] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:13.893159 16814 raft_consensus.cc:729] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:13.896149 16907 leader_election.cc:288] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:13.896103 16814 consensus_queue.cc:260] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.902400 16958 leader_election.cc:288] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:13.907635 16958 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:13.911648 16958 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.916724 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:13.918354 16746 raft_consensus.cc:2454] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 0.
I20260810 18:51:13.923841 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:13.924261 16958 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:13.925099 16505 raft_consensus.cc:2379] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 1: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:13.926419 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:13.933789 16450 leader_election.cc:302] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:13.933468 16817 tablet_bootstrap.cc:492] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:13.937286 16817 ts_tablet_manager.cc:1397] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.177s user 0.011s sys 0.015s
I20260810 18:51:13.937332 17196 raft_consensus.cc:2781] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:13.938016 16814 ts_tablet_manager.cc:1428] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.119s user 0.006s sys 0.004s
I20260810 18:51:13.937959 17196 raft_consensus.cc:482] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:13.939589 17196 raft_consensus.cc:3037] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:13.939662 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:13.940394 16907 consensus_queue.cc:1035] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:13.943264 16748 raft_consensus.cc:2454] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:13.946633 16818 ts_tablet_manager.cc:1428] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.123s user 0.009s sys 0.003s
W20260810 18:51:13.946667 16448 leader_election.cc:341] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:13.942790 16817 raft_consensus.cc:348] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.950062 16817 raft_consensus.cc:374] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:13.950412 16817 raft_consensus.cc:729] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:13.949486 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:13.951673 16450 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:13.954537 16907 raft_consensus.cc:2781] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:13.955137 16907 raft_consensus.cc:686] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:13.955648 17332 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.956871 16818 tablet_bootstrap.cc:492] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:13.956444 16907 consensus_queue.cc:237] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.966269 16814 tablet_bootstrap.cc:492] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:13.966341 16505 raft_consensus.cc:3037] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:13.967095 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:13.974767 17196 raft_consensus.cc:504] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.977320 17196 leader_election.cc:288] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:13.955149 16817 consensus_queue.cc:260] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:13.982115 17331 consensus_queue.cc:1035] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:13.983665 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:13.986874 16701 leader_election.cc:302] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:13.988479 16817 ts_tablet_manager.cc:1428] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.050s user 0.007s sys 0.001s
I20260810 18:51:13.989166 17332 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:13.990236 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:13.991087 16507 raft_consensus.cc:3037] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:13.991778 17350 raft_consensus.cc:3037] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:13.994583 16817 tablet_bootstrap.cc:492] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
W20260810 18:51:13.998252 16695 leader_election.cc:341] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
W20260810 18:51:13.998674 16448 leader_election.cc:341] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:13.994937 17333 consensus_queue.cc:1035] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260810 18:51:14.005055 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:14.005749 16750 raft_consensus.cc:3037] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
W20260810 18:51:14.010674 16701 leader_election.cc:341] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:14.011238 16701 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:14.016621 17353 raft_consensus.cc:2726] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:14.017258 16818 tablet_bootstrap.cc:654] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:14.024137 16750 raft_consensus.cc:2454] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:14.037521 17196 consensus_queue.cc:1035] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:14.043907 16814 tablet_bootstrap.cc:654] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:14.055732 16450 leader_election.cc:302] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:14.056445 17350 raft_consensus.cc:2726] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:14.058271 17245 raft_consensus.cc:2781] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:14.058835 17245 raft_consensus.cc:686] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:14.057096 16338 catalog_manager.cc:5495] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 0 to 1, leader changed from <none> to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:14.059862 17245 consensus_queue.cc:237] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.073980 17332 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:14.065054 16507 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:14.074571 17332 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.079692 16817 tablet_bootstrap.cc:654] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:14.071056 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:14.091936 17332 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:14.101511 16750 raft_consensus.cc:3037] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:14.105607 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:14.098958 17333 consensus_queue.cc:1035] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260810 18:51:14.111455 16749 raft_consensus.cc:2379] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 1: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:14.105623 16573 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:14.114305 17333 raft_consensus.cc:2781] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:14.118834 17333 raft_consensus.cc:686] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:14.124725 17333 consensus_queue.cc:237] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.141695 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:14.158164 16750 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
W20260810 18:51:14.196025 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 105951us
I20260810 18:51:14.271763 16818 tablet_bootstrap.cc:492] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:14.272550 16818 ts_tablet_manager.cc:1397] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.316s user 0.016s sys 0.008s
I20260810 18:51:14.273818 16342 catalog_manager.cc:5495] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:14.276362 16817 tablet_bootstrap.cc:492] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:14.276932 16817 ts_tablet_manager.cc:1397] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.287s user 0.035s sys 0.042s
W20260810 18:51:14.292618 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 134532us
I20260810 18:51:14.296909 16573 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:14.297382 16338 catalog_manager.cc:5495] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 0 to 1, leader changed from <none> to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:14.280030 16817 raft_consensus.cc:348] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.307045 16817 raft_consensus.cc:374] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:14.307461 16817 raft_consensus.cc:729] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:14.308331 16814 tablet_bootstrap.cc:492] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:14.308965 16814 ts_tablet_manager.cc:1397] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.343s user 0.007s sys 0.010s
I20260810 18:51:14.315186 16817 consensus_queue.cc:260] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:14.321449 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 51036.8us
I20260810 18:51:14.324177 17333 raft_consensus.cc:3037] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:14.330384 16817 ts_tablet_manager.cc:1428] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.053s user 0.008s sys 0.000s
I20260810 18:51:14.332114 16818 raft_consensus.cc:348] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.333243 16818 raft_consensus.cc:374] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:14.333539 16818 raft_consensus.cc:729] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:14.337352 16814 raft_consensus.cc:348] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.337929 16814 raft_consensus.cc:374] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:14.338192 16814 raft_consensus.cc:729] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:14.340662 16814 consensus_queue.cc:260] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.347342 16818 consensus_queue.cc:260] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.349828 16818 ts_tablet_manager.cc:1428] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.077s user 0.003s sys 0.001s
I20260810 18:51:14.351034 16818 tablet_bootstrap.cc:492] T 192ad39135ed48d5adab628ba706791a P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:14.357017 16817 tablet_bootstrap.cc:492] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:14.362994 17333 raft_consensus.cc:2726] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:14.380124 16814 ts_tablet_manager.cc:1428] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.071s user 0.004s sys 0.004s
I20260810 18:51:14.380330 17199 consensus_queue.cc:1035] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:14.381232 16814 tablet_bootstrap.cc:492] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:14.407171 16818 tablet_bootstrap.cc:654] T 192ad39135ed48d5adab628ba706791a P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:14.442655 17362 consensus_queue.cc:1035] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:14.444149 16817 tablet_bootstrap.cc:654] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:14.451309 16814 tablet_bootstrap.cc:654] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:14.453279 16958 raft_consensus.cc:482] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:14.453840 16958 raft_consensus.cc:504] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.456056 16958 leader_election.cc:288] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:14.457643 16818 tablet_bootstrap.cc:492] T 192ad39135ed48d5adab628ba706791a P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:14.458316 16818 ts_tablet_manager.cc:1397] T 192ad39135ed48d5adab628ba706791a P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.108s user 0.005s sys 0.009s
I20260810 18:51:14.460485 17025 raft_consensus.cc:482] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:14.461977 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:14.461465 17025 raft_consensus.cc:504] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.463338 16814 tablet_bootstrap.cc:492] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:14.463932 16814 ts_tablet_manager.cc:1397] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.083s user 0.005s sys 0.010s
I20260810 18:51:14.463372 16818 raft_consensus.cc:348] T 192ad39135ed48d5adab628ba706791a P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.464259 16818 raft_consensus.cc:374] T 192ad39135ed48d5adab628ba706791a P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:14.464529 16818 raft_consensus.cc:729] T 192ad39135ed48d5adab628ba706791a P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:14.464744 17025 leader_election.cc:288] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:14.465121 16818 consensus_queue.cc:260] T 192ad39135ed48d5adab628ba706791a P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.466321 16814 raft_consensus.cc:348] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.467424 16814 raft_consensus.cc:374] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:14.467681 16814 raft_consensus.cc:729] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:14.468379 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:14.469703 16701 leader_election.cc:341] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:14.469961 16907 consensus_queue.cc:1035] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:14.470839 16814 consensus_queue.cc:260] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:14.471592 16695 leader_election.cc:341] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:14.472158 16695 leader_election.cc:302] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:14.471531 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:14.473543 16814 ts_tablet_manager.cc:1428] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260810 18:51:14.473899 17048 raft_consensus.cc:2726] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:14.474515 16814 tablet_bootstrap.cc:492] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:14.474951 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:14.475934 16504 raft_consensus.cc:2454] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 0.
I20260810 18:51:14.476158 16818 ts_tablet_manager.cc:1428] T 192ad39135ed48d5adab628ba706791a P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.017s user 0.006s sys 0.003s
W20260810 18:51:14.478267 16701 leader_election.cc:341] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:14.478389 16818 tablet_bootstrap.cc:492] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:14.480718 17372 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:14.481166 17372 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.482985 16814 tablet_bootstrap.cc:654] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:14.485286 16817 tablet_bootstrap.cc:492] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:14.485749 16817 ts_tablet_manager.cc:1397] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.129s user 0.012s sys 0.008s
I20260810 18:51:14.488790 16817 raft_consensus.cc:348] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.489428 16817 raft_consensus.cc:374] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:14.489699 16817 raft_consensus.cc:729] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:14.490289 16817 consensus_queue.cc:260] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.493369 16695 leader_election.cc:302] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:14.496037 16907 consensus_queue.cc:1035] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:51:14.496254 17372 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:14.499079 17048 raft_consensus.cc:2781] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:14.499442 17048 raft_consensus.cc:482] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:14.499786 17048 raft_consensus.cc:3037] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:14.498857 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:14.501739 16814 tablet_bootstrap.cc:492] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:14.504671 16814 ts_tablet_manager.cc:1397] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.000s
W20260810 18:51:14.505744 16448 leader_election.cc:341] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:14.514307 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:14.515587 16817 ts_tablet_manager.cc:1428] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.030s user 0.005s sys 0.004s
I20260810 18:51:14.518491 16818 tablet_bootstrap.cc:654] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:14.519122 16817 tablet_bootstrap.cc:492] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:14.528612 17048 raft_consensus.cc:504] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.530967 16814 raft_consensus.cc:348] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.534262 16814 raft_consensus.cc:374] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:14.534617 16814 raft_consensus.cc:729] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:14.535338 16814 consensus_queue.cc:260] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.537686 17384 raft_consensus.cc:482] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:14.538136 17384 raft_consensus.cc:504] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:14.542085 16450 leader_election.cc:341] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:14.542514 16450 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:14.542570 16814 ts_tablet_manager.cc:1428] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.037s user 0.007s sys 0.000s
I20260810 18:51:14.544425 16814 tablet_bootstrap.cc:492] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:14.545511 17195 raft_consensus.cc:2726] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:14.548740 17384 leader_election.cc:288] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:14.550827 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:14.553910 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:14.554769 16633 raft_consensus.cc:3037] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:14.555799 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:14.556404 16504 raft_consensus.cc:3037] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
W20260810 18:51:14.558840 16701 leader_election.cc:341] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:14.569903 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:14.569908 17121 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:14.550798 17048 leader_election.cc:288] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:14.578827 16817 tablet_bootstrap.cc:654] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:14.579291 17121 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.580559 16575 leader_election.cc:302] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:14.584713 16814 tablet_bootstrap.cc:654] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:14.592674 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:14.593427 16505 raft_consensus.cc:2454] T 192ad39135ed48d5adab628ba706791a P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 0.
I20260810 18:51:14.594960 16573 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:14.595157 17121 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:14.595718 17121 raft_consensus.cc:2781] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:14.596086 17121 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:14.596426 17121 raft_consensus.cc:3037] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:14.603240 16504 raft_consensus.cc:2454] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:14.608865 17331 consensus_queue.cc:1035] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:51:14.609506 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:14.613386 16695 leader_election.cc:302] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:14.615366 17025 raft_consensus.cc:2781] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:14.615861 17025 raft_consensus.cc:686] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:14.616696 17025 consensus_queue.cc:237] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:14.620529 16575 leader_election.cc:341] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:14.641702 17384 raft_consensus.cc:2726] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:14.643471 16818 tablet_bootstrap.cc:492] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:14.644127 16818 ts_tablet_manager.cc:1397] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.167s user 0.029s sys 0.017s
I20260810 18:51:14.647362 17195 consensus_queue.cc:1035] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:51:14.650781 16818 raft_consensus.cc:348] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.656230 16814 tablet_bootstrap.cc:492] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:14.656852 16814 ts_tablet_manager.cc:1397] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.113s user 0.004s sys 0.018s
I20260810 18:51:14.651728 16818 raft_consensus.cc:374] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:14.658962 16818 raft_consensus.cc:729] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:14.659996 16818 consensus_queue.cc:260] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.662791 17384 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:14.663398 17384 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.662734 16342 catalog_manager.cc:5495] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 0 to 1, leader changed from <none> to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:51:14.665324 17384 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:14.665370 16814 raft_consensus.cc:348] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.666183 16814 raft_consensus.cc:374] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:14.666450 16814 raft_consensus.cc:729] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:14.667374 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:14.667770 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:14.668253 16505 raft_consensus.cc:2454] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 0.
I20260810 18:51:14.667416 16814 consensus_queue.cc:260] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.671613 16573 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:51:14.672282 16575 leader_election.cc:341] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:14.684670 17386 raft_consensus.cc:2781] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:14.685056 17386 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:14.685345 17386 raft_consensus.cc:3037] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:14.686704 16814 ts_tablet_manager.cc:1428] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.030s user 0.008s sys 0.000s
I20260810 18:51:14.689968 16818 ts_tablet_manager.cc:1428] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.045s user 0.009s sys 0.000s
I20260810 18:51:14.689877 17121 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.693806 17121 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:14.694525 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:14.695206 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:14.695216 16505 raft_consensus.cc:3037] T 192ad39135ed48d5adab628ba706791a P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
W20260810 18:51:14.697214 16575 leader_election.cc:341] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:14.698577 16817 tablet_bootstrap.cc:492] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:14.699107 16817 ts_tablet_manager.cc:1397] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.180s user 0.030s sys 0.033s
I20260810 18:51:14.701973 16817 raft_consensus.cc:348] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.702638 16817 raft_consensus.cc:374] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:14.695412 17121 consensus_queue.cc:1035] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260810 18:51:14.702884 16817 raft_consensus.cc:729] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:14.705374 16817 consensus_queue.cc:260] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.708019 16817 ts_tablet_manager.cc:1428] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260810 18:51:14.708703 16814 tablet_bootstrap.cc:492] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:14.709087 16817 tablet_bootstrap.cc:492] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:14.714958 16818 tablet_bootstrap.cc:492] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:14.702994 16505 raft_consensus.cc:2454] T 192ad39135ed48d5adab628ba706791a P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:14.719751 16573 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:14.722647 17121 raft_consensus.cc:2781] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:14.723142 17121 raft_consensus.cc:686] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:14.723922 17121 consensus_queue.cc:237] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.730317 17386 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.735556 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:14.736364 16505 raft_consensus.cc:3037] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:14.773440 16818 tablet_bootstrap.cc:654] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:14.776580 17025 raft_consensus.cc:482] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:14.777087 17025 raft_consensus.cc:504] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.777683 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
W20260810 18:51:14.779387 16575 leader_election.cc:341] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:14.779662 17025 leader_election.cc:288] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:14.780915 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:51:14.782398 16695 leader_election.cc:341] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:14.780915 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:14.784598 16701 leader_election.cc:341] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:14.784947 16701 leader_election.cc:302] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:14.782047 17386 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:14.787652 17025 raft_consensus.cc:2726] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:14.789431 16505 raft_consensus.cc:2454] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:14.790560 16573 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:14.790763 16817 tablet_bootstrap.cc:654] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:14.791973 16814 tablet_bootstrap.cc:654] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:14.792443 17333 raft_consensus.cc:2781] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:14.797221 17333 raft_consensus.cc:686] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:14.799489 17333 consensus_queue.cc:237] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.800779 16818 tablet_bootstrap.cc:492] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:14.801242 16818 ts_tablet_manager.cc:1397] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.087s user 0.010s sys 0.015s
I20260810 18:51:14.803813 16345 catalog_manager.cc:5495] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:14.808194 16818 raft_consensus.cc:348] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.809008 16818 raft_consensus.cc:374] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:14.809275 16818 raft_consensus.cc:729] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:14.809998 16818 consensus_queue.cc:260] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.813161 16817 tablet_bootstrap.cc:492] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:14.815158 16817 ts_tablet_manager.cc:1397] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.106s user 0.015s sys 0.012s
I20260810 18:51:14.819753 16817 raft_consensus.cc:348] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.822835 16817 raft_consensus.cc:374] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:14.823197 16817 raft_consensus.cc:729] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:14.827451 16818 ts_tablet_manager.cc:1428] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.026s user 0.005s sys 0.003s
W20260810 18:51:14.833153 16415 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260810 18:51:14.835284 16817 consensus_queue.cc:260] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.837132 16342 catalog_manager.cc:5495] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:14.838132 16817 ts_tablet_manager.cc:1428] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20260810 18:51:14.839788 16817 tablet_bootstrap.cc:492] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:14.842923 16814 tablet_bootstrap.cc:492] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:14.843874 16818 tablet_bootstrap.cc:492] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:14.844892 16814 ts_tablet_manager.cc:1397] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.136s user 0.020s sys 0.020s
I20260810 18:51:14.857220 16817 tablet_bootstrap.cc:654] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:14.863245 16818 tablet_bootstrap.cc:654] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:14.866693 16814 raft_consensus.cc:348] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.868386 16814 raft_consensus.cc:374] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:14.868499 16818 tablet_bootstrap.cc:492] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:14.868709 16814 raft_consensus.cc:729] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:14.868930 16818 ts_tablet_manager.cc:1397] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.003s
I20260810 18:51:14.869415 16814 consensus_queue.cc:260] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.871613 16814 ts_tablet_manager.cc:1428] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20260810 18:51:14.871353 16818 raft_consensus.cc:348] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.872044 16818 raft_consensus.cc:374] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:14.872326 16818 raft_consensus.cc:729] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:14.872933 16818 consensus_queue.cc:260] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.873724 16817 tablet_bootstrap.cc:492] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:14.874233 16817 ts_tablet_manager.cc:1397] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.005s
I20260810 18:51:14.875137 16818 ts_tablet_manager.cc:1428] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 18:51:14.876163 16818 tablet_bootstrap.cc:492] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:14.878183 16814 tablet_bootstrap.cc:492] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:14.881837 16818 tablet_bootstrap.cc:654] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:14.882133 16817 raft_consensus.cc:348] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.882907 16817 raft_consensus.cc:374] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:14.883253 16817 raft_consensus.cc:729] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:14.883973 16817 consensus_queue.cc:260] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.886268 16817 ts_tablet_manager.cc:1428] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260810 18:51:14.899428 16817 tablet_bootstrap.cc:492] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:14.902290 16814 tablet_bootstrap.cc:654] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:14.906038 16817 tablet_bootstrap.cc:654] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:14.908833 16958 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:14.909334 16958 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.913846 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:14.914678 16504 raft_consensus.cc:2454] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 0.
I20260810 18:51:14.918303 16695 leader_election.cc:302] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:14.919382 17349 raft_consensus.cc:2781] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:14.919839 17349 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:14.920245 17349 raft_consensus.cc:3037] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:14.922190 16818 tablet_bootstrap.cc:492] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:14.922677 16818 ts_tablet_manager.cc:1397] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.047s user 0.007s sys 0.014s
I20260810 18:51:14.925441 16818 raft_consensus.cc:348] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.926188 16818 raft_consensus.cc:374] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:14.926463 16818 raft_consensus.cc:729] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:14.927160 16818 consensus_queue.cc:260] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.930706 16814 tablet_bootstrap.cc:492] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:14.931212 16814 ts_tablet_manager.cc:1397] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.053s user 0.008s sys 0.006s
I20260810 18:51:14.932698 17386 raft_consensus.cc:482] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:14.933303 16817 tablet_bootstrap.cc:492] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:14.933133 17386 raft_consensus.cc:504] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.933720 16817 ts_tablet_manager.cc:1397] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.012s
I20260810 18:51:14.948422 17386 leader_election.cc:288] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:14.932232 16958 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:14.930501 16818 ts_tablet_manager.cc:1428] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260810 18:51:14.948622 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:14.949420 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:14.949450 16504 raft_consensus.cc:2454] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 0.
I20260810 18:51:14.950047 16750 raft_consensus.cc:2454] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 0.
I20260810 18:51:14.950614 16573 leader_election.cc:302] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:14.936244 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:14.947883 16814 raft_consensus.cc:348] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.952143 16814 raft_consensus.cc:374] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:14.952783 16818 tablet_bootstrap.cc:492] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:14.955415 16814 raft_consensus.cc:729] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:14.948639 16817 raft_consensus.cc:348] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.958020 16817 raft_consensus.cc:374] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:14.952800 17386 raft_consensus.cc:2781] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:14.959824 17386 raft_consensus.cc:482] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:14.965044 17386 raft_consensus.cc:3037] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
W20260810 18:51:14.960151 16701 leader_election.cc:341] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:14.958143 16814 consensus_queue.cc:260] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.958344 16817 raft_consensus.cc:729] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:14.964689 16818 tablet_bootstrap.cc:654] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:14.953190 17349 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.976174 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:14.976780 16504 raft_consensus.cc:3037] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:14.977902 16814 ts_tablet_manager.cc:1428] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.046s user 0.005s sys 0.001s
I20260810 18:51:14.978904 16814 tablet_bootstrap.cc:492] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:14.980876 16817 consensus_queue.cc:260] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.981953 17349 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:14.982908 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:51:14.984304 16701 leader_election.cc:341] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:14.990182 16817 ts_tablet_manager.cc:1428] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.044s user 0.005s sys 0.001s
I20260810 18:51:14.987668 16504 raft_consensus.cc:2454] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:14.992040 16695 leader_election.cc:302] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:14.993124 17350 raft_consensus.cc:2781] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:14.993577 17350 raft_consensus.cc:686] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:14.994601 16814 tablet_bootstrap.cc:654] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:14.994357 17350 consensus_queue.cc:237] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.995836 17362 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:14.996414 17362 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:14.999995 17362 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:15.001435 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:15.003713 16448 leader_election.cc:341] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:15.005574 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:15.006485 16750 raft_consensus.cc:2454] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 0.
I20260810 18:51:15.008091 16450 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:15.011425 16817 tablet_bootstrap.cc:492] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:15.011605 17362 raft_consensus.cc:2781] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:15.012109 17362 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:15.012488 17362 raft_consensus.cc:3037] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:15.025449 17386 raft_consensus.cc:504] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.027752 16958 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:15.028244 16958 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.030681 16958 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:15.034866 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:15.036680 16750 raft_consensus.cc:3037] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:15.037029 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:15.036401 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:15.031215 17386 leader_election.cc:288] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:15.037803 16633 raft_consensus.cc:2454] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 0.
W20260810 18:51:15.040333 16695 leader_election.cc:341] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:15.040940 16701 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:15.040742 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:15.042935 17350 raft_consensus.cc:2781] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:15.043143 16504 raft_consensus.cc:3037] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:15.046739 16750 raft_consensus.cc:2454] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:15.047762 16575 leader_election.cc:302] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:15.048822 16818 tablet_bootstrap.cc:492] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:15.049394 16818 ts_tablet_manager.cc:1397] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.097s user 0.020s sys 0.027s
I20260810 18:51:15.049707 17350 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:15.049840 17386 raft_consensus.cc:2781] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:15.052376 17350 raft_consensus.cc:3037] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:15.053465 16344 catalog_manager.cc:5495] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 0 to 1, leader changed from <none> to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:51:15.054801 17362 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.057544 16818 raft_consensus.cc:348] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.057896 17386 raft_consensus.cc:686] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:15.055279 16504 raft_consensus.cc:2454] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:15.058779 17386 consensus_queue.cc:237] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.059115 16818 raft_consensus.cc:374] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:15.061118 16818 raft_consensus.cc:729] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:15.061784 16818 consensus_queue.cc:260] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.062160 17362 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:15.058604 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:15.062924 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:15.063036 16817 tablet_bootstrap.cc:654] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:15.116888 16958 consensus_queue.cc:1035] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:15.066732 17350 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:15.067554 16448 leader_election.cc:341] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
W20260810 18:51:15.119706 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 52217.6us
I20260810 18:51:15.121191 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
W20260810 18:51:15.122548 16695 leader_election.cc:341] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:15.064731 16818 ts_tablet_manager.cc:1428] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.015s user 0.005s sys 0.001s
I20260810 18:51:15.123039 16344 catalog_manager.cc:5495] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:15.130079 17350 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:15.130223 16818 tablet_bootstrap.cc:492] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
W20260810 18:51:15.125007 16701 consensus_peers.cc:487] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be -> Peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Couldn't send request to peer 0397d760589d40d3b92ef141cdfc2dbb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:15.134239 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:15.134833 16633 raft_consensus.cc:3037] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:15.140040 16750 raft_consensus.cc:3037] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:15.144594 16814 tablet_bootstrap.cc:492] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:15.144768 16633 raft_consensus.cc:2454] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:15.145073 16814 ts_tablet_manager.cc:1397] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.166s user 0.004s sys 0.007s
I20260810 18:51:15.139961 17195 raft_consensus.cc:482] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:15.146001 17195 raft_consensus.cc:504] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.147650 16814 raft_consensus.cc:348] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.148373 16814 raft_consensus.cc:374] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:15.148671 16814 raft_consensus.cc:729] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:15.149264 16814 consensus_queue.cc:260] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.149521 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:15.149955 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:15.150207 16746 raft_consensus.cc:2454] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 0.
W20260810 18:51:15.151274 16448 leader_election.cc:341] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:15.151535 17195 leader_election.cc:288] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:15.151587 16814 ts_tablet_manager.cc:1428] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260810 18:51:15.152185 16450 leader_election.cc:302] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:15.153205 16818 tablet_bootstrap.cc:654] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:15.153940 17372 raft_consensus.cc:2781] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:15.154343 17372 raft_consensus.cc:482] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:15.154663 17372 raft_consensus.cc:3037] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:15.151590 16701 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:15.165990 17372 raft_consensus.cc:504] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.167183 16814 tablet_bootstrap.cc:492] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:15.169006 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:15.183678 17386 consensus_queue.cc:1035] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 18:51:15.185030 16448 leader_election.cc:341] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:15.188766 17384 raft_consensus.cc:482] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:15.189235 17384 raft_consensus.cc:504] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.190176 16814 tablet_bootstrap.cc:654] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:15.171697 16958 raft_consensus.cc:2781] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:15.169005 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:15.196391 17384 leader_election.cc:288] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:15.197640 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:15.198352 16958 raft_consensus.cc:686] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:15.170558 17372 leader_election.cc:288] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:15.218798 16818 tablet_bootstrap.cc:492] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:15.223312 16818 ts_tablet_manager.cc:1397] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.093s user 0.028s sys 0.013s
I20260810 18:51:15.220582 16749 raft_consensus.cc:3037] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:15.229455 16817 tablet_bootstrap.cc:492] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:15.221930 16746 raft_consensus.cc:2454] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 0.
I20260810 18:51:15.229907 16958 consensus_queue.cc:237] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.229955 16817 ts_tablet_manager.cc:1397] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.219s user 0.016s sys 0.010s
I20260810 18:51:15.234545 16575 leader_election.cc:302] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:15.234687 16752 raft_consensus.cc:3037] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:15.246351 17384 raft_consensus.cc:2781] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:15.246893 16817 raft_consensus.cc:348] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.248757 16817 raft_consensus.cc:374] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:15.249050 16817 raft_consensus.cc:729] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:15.249732 16817 consensus_queue.cc:260] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.252240 16817 ts_tablet_manager.cc:1428] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260810 18:51:15.253244 16817 tablet_bootstrap.cc:492] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:15.256100 17452 raft_consensus.cc:482] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:15.246874 17384 raft_consensus.cc:482] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:15.256971 17384 raft_consensus.cc:3037] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:15.256637 17452 raft_consensus.cc:504] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.242991 16749 raft_consensus.cc:2454] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:15.263391 17386 consensus_queue.cc:1035] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:51:15.265134 16450 leader_election.cc:302] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:15.267097 17451 raft_consensus.cc:2781] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:15.267607 17451 raft_consensus.cc:686] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:15.268625 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:15.268481 17451 consensus_queue.cc:237] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.269266 16636 raft_consensus.cc:2454] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 0.
I20260810 18:51:15.271327 16448 leader_election.cc:302] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:15.272112 17372 raft_consensus.cc:2781] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:15.272471 17372 raft_consensus.cc:482] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:15.272811 17372 raft_consensus.cc:3037] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:15.275632 17452 leader_election.cc:288] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:15.276230 16818 raft_consensus.cc:348] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.276890 16818 raft_consensus.cc:374] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:15.277137 16818 raft_consensus.cc:729] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:15.279402 16818 consensus_queue.cc:260] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.277078 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:15.283692 16750 raft_consensus.cc:2454] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
W20260810 18:51:15.286182 16450 leader_election.cc:341] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:15.287041 16450 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:15.288919 17362 raft_consensus.cc:2781] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:15.289980 17362 raft_consensus.cc:686] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:15.290761 17362 consensus_queue.cc:237] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.295197 16814 tablet_bootstrap.cc:492] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:15.295661 16814 ts_tablet_manager.cc:1397] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.142s user 0.015s sys 0.012s
I20260810 18:51:15.312731 16814 raft_consensus.cc:348] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.313429 16814 raft_consensus.cc:374] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:15.313686 16814 raft_consensus.cc:729] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:15.314281 16814 consensus_queue.cc:260] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:15.314723 16575 consensus_peers.cc:487] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb -> Peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Couldn't send request to peer 980f343dd1fd4e049143f8b9d9f2f1be. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:15.313894 16342 catalog_manager.cc:5495] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 0 to 1, leader changed from <none> to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:51:15.318084 16818 ts_tablet_manager.cc:1428] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.094s user 0.006s sys 0.000s
I20260810 18:51:15.318223 16817 tablet_bootstrap.cc:654] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:15.328585 16814 ts_tablet_manager.cc:1428] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.033s user 0.006s sys 0.000s
I20260810 18:51:15.317997 17384 raft_consensus.cc:504] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.340242 16344 catalog_manager.cc:5495] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 0 to 1, leader changed from <none> to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:15.344622 16814 tablet_bootstrap.cc:492] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:15.362404 16818 tablet_bootstrap.cc:492] T abdb4e808ce3454ab9a52051b369e5d0 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:15.367244 16958 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:15.386459 17384 leader_election.cc:288] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:15.395082 17386 raft_consensus.cc:482] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:15.392421 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:15.395563 17386 raft_consensus.cc:504] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.397730 17386 leader_election.cc:288] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:15.393922 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:15.400565 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:15.401662 16750 raft_consensus.cc:3037] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:15.406622 17372 raft_consensus.cc:504] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.385228 16958 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.420688 16814 tablet_bootstrap.cc:654] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:15.435654 16958 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:15.436807 16818 tablet_bootstrap.cc:654] T abdb4e808ce3454ab9a52051b369e5d0 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:15.438086 16817 tablet_bootstrap.cc:492] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:15.438508 16817 ts_tablet_manager.cc:1397] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.185s user 0.015s sys 0.002s
I20260810 18:51:15.440218 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:15.440235 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:15.440692 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:51:15.441987 16695 leader_election.cc:341] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
W20260810 18:51:15.444796 16701 leader_election.cc:341] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:15.446305 17372 leader_election.cc:288] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:15.445549 16701 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:15.447763 16958 raft_consensus.cc:2726] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:15.450605 16750 raft_consensus.cc:2454] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:15.470916 16817 raft_consensus.cc:348] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.471724 16817 raft_consensus.cc:374] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:15.471966 16817 raft_consensus.cc:729] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:15.472543 16817 consensus_queue.cc:260] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.473951 16817 ts_tablet_manager.cc:1428] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.035s user 0.003s sys 0.003s
I20260810 18:51:15.474874 16817 tablet_bootstrap.cc:492] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:15.479553 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:15.476814 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:15.479584 16344 catalog_manager.cc:5495] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 0 to 1, leader changed from <none> to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:15.493862 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:51:15.494100 17463 log.cc:924] Time spent T 192ad39135ed48d5adab628ba706791a P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.070s user 0.001s sys 0.000s
I20260810 18:51:15.507752 16636 raft_consensus.cc:3037] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:15.508097 16749 raft_consensus.cc:3037] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:15.512562 16818 tablet_bootstrap.cc:492] T abdb4e808ce3454ab9a52051b369e5d0 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:15.513113 16818 ts_tablet_manager.cc:1397] T abdb4e808ce3454ab9a52051b369e5d0 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.177s user 0.013s sys 0.000s
I20260810 18:51:15.514779 16575 leader_election.cc:302] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
W20260810 18:51:15.518952 16573 leader_election.cc:341] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:15.520710 16749 raft_consensus.cc:2454] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:15.516611 17384 raft_consensus.cc:2781] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:15.521665 17384 raft_consensus.cc:686] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:15.522496 17384 consensus_queue.cc:237] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.523947 16450 leader_election.cc:302] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:15.524745 17362 raft_consensus.cc:2781] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:15.525231 17362 raft_consensus.cc:686] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:15.526036 17362 consensus_queue.cc:237] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.527340 16636 raft_consensus.cc:2454] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:15.531652 16814 tablet_bootstrap.cc:492] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:15.532922 16814 ts_tablet_manager.cc:1397] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.197s user 0.009s sys 0.005s
I20260810 18:51:15.539829 16814 raft_consensus.cc:348] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.540719 16814 raft_consensus.cc:374] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:15.541078 16814 raft_consensus.cc:729] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:15.541898 16814 consensus_queue.cc:260] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.545061 16814 ts_tablet_manager.cc:1428] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.012s user 0.004s sys 0.001s
I20260810 18:51:15.546350 16814 tablet_bootstrap.cc:492] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:15.550205 16817 tablet_bootstrap.cc:654] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:15.550628 16818 raft_consensus.cc:348] T abdb4e808ce3454ab9a52051b369e5d0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.551616 16818 raft_consensus.cc:374] T abdb4e808ce3454ab9a52051b369e5d0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:15.551951 16818 raft_consensus.cc:729] T abdb4e808ce3454ab9a52051b369e5d0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:15.552675 16818 consensus_queue.cc:260] T abdb4e808ce3454ab9a52051b369e5d0 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.566193 17025 consensus_queue.cc:1035] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:51:15.569796 16818 ts_tablet_manager.cc:1428] T abdb4e808ce3454ab9a52051b369e5d0 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.056s user 0.007s sys 0.000s
I20260810 18:51:15.571112 16818 tablet_bootstrap.cc:492] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:15.579839 16636 raft_consensus.cc:3037] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:15.580292 16345 catalog_manager.cc:5495] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
W20260810 18:51:15.581465 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 52338.5us
W20260810 18:51:15.589691 16573 leader_election.cc:341] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:15.592504 17386 consensus_queue.cc:1035] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:15.596246 17121 consensus_queue.cc:1035] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:51:15.604971 16342 catalog_manager.cc:5495] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 0 to 1, leader changed from <none> to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:15.615801 16814 tablet_bootstrap.cc:654] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:15.615844 16633 raft_consensus.cc:3037] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:15.634769 16817 tablet_bootstrap.cc:492] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:15.635444 16817 ts_tablet_manager.cc:1397] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.161s user 0.021s sys 0.018s
I20260810 18:51:15.640206 16817 raft_consensus.cc:348] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.640966 16817 raft_consensus.cc:374] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:15.641253 16817 raft_consensus.cc:729] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:15.646147 16817 consensus_queue.cc:260] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.624099 16573 leader_election.cc:302] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:15.656697 16817 ts_tablet_manager.cc:1428] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.021s user 0.005s sys 0.001s
I20260810 18:51:15.659981 16814 tablet_bootstrap.cc:492] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:15.659953 17384 consensus_queue.cc:1035] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:15.660895 16818 tablet_bootstrap.cc:654] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:15.662770 16814 ts_tablet_manager.cc:1397] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.117s user 0.020s sys 0.000s
I20260810 18:51:15.679953 17386 raft_consensus.cc:2726] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:15.701071 17333 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:15.701663 17333 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.707525 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:15.707600 17025 consensus_queue.cc:1035] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:15.708526 17333 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:15.710928 16817 tablet_bootstrap.cc:492] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:15.720945 16814 raft_consensus.cc:348] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.721675 16818 tablet_bootstrap.cc:492] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:15.722162 16818 ts_tablet_manager.cc:1397] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.151s user 0.020s sys 0.000s
I20260810 18:51:15.724534 16818 raft_consensus.cc:348] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.725199 16818 raft_consensus.cc:374] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:15.725453 16818 raft_consensus.cc:729] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:15.721721 16814 raft_consensus.cc:374] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:15.726143 16814 raft_consensus.cc:729] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:15.726780 16814 consensus_queue.cc:260] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.730973 16814 ts_tablet_manager.cc:1428] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.068s user 0.007s sys 0.000s
I20260810 18:51:15.735505 16814 tablet_bootstrap.cc:492] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:15.741389 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:15.748042 16818 consensus_queue.cc:260] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.754799 16817 tablet_bootstrap.cc:654] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:15.758350 16573 leader_election.cc:302] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:15.763139 16818 ts_tablet_manager.cc:1428] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.041s user 0.009s sys 0.000s
I20260810 18:51:15.770841 17450 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:15.771169 16818 tablet_bootstrap.cc:492] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:15.771428 17450 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.773916 17450 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:15.777814 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:15.778666 16504 raft_consensus.cc:3037] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:15.812161 16814 tablet_bootstrap.cc:654] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:15.840185 17333 raft_consensus.cc:2726] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 18:51:15.844415 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 60688.6us
I20260810 18:51:15.846660 16958 consensus_queue.cc:1035] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:51:15.858455 16958 consensus_queue.cc:1035] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:51:15.860285 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:15.866145 16814 tablet_bootstrap.cc:492] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:15.866632 16814 ts_tablet_manager.cc:1397] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.131s user 0.011s sys 0.006s
I20260810 18:51:15.869141 16818 tablet_bootstrap.cc:654] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:15.912147 16817 tablet_bootstrap.cc:492] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:15.912752 16817 ts_tablet_manager.cc:1397] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.247s user 0.016s sys 0.007s
I20260810 18:51:15.910670 16814 raft_consensus.cc:348] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.913329 16814 raft_consensus.cc:374] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:15.913722 16814 raft_consensus.cc:729] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:15.914592 16814 consensus_queue.cc:260] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.911185 17362 raft_consensus.cc:482] T abdb4e808ce3454ab9a52051b369e5d0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:15.916090 17362 raft_consensus.cc:504] T abdb4e808ce3454ab9a52051b369e5d0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.918139 17362 leader_election.cc:288] T abdb4e808ce3454ab9a52051b369e5d0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:15.925916 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:15.929860 16817 raft_consensus.cc:348] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.915400 16448 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:15.930593 16817 raft_consensus.cc:374] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:15.933910 16814 ts_tablet_manager.cc:1428] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.067s user 0.007s sys 0.000s
I20260810 18:51:15.934877 16814 tablet_bootstrap.cc:492] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
W20260810 18:51:15.936990 16450 leader_election.cc:341] T abdb4e808ce3454ab9a52051b369e5d0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:15.931118 16817 raft_consensus.cc:729] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:15.940104 16817 consensus_queue.cc:260] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.956907 16817 ts_tablet_manager.cc:1428] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.044s user 0.006s sys 0.001s
W20260810 18:51:15.960117 16415 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260810 18:51:15.970049 17475 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:51:15.961205 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 45895.7us
I20260810 18:51:15.970724 17475 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:15.971773 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:15.961871 17457 raft_consensus.cc:2726] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:15.973150 17475 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:15.975135 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:15.975826 16750 raft_consensus.cc:2379] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 1: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:15.975589 17451 consensus_queue.cc:1035] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:51:15.977252 16626 raft_consensus.cc:3037] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:15.980650 16814 tablet_bootstrap.cc:654] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:15.983444 16818 tablet_bootstrap.cc:492] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:15.983907 16818 ts_tablet_manager.cc:1397] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.213s user 0.026s sys 0.026s
I20260810 18:51:15.986284 16817 tablet_bootstrap.cc:492] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:15.991968 17457 consensus_queue.cc:1035] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:16.002481 16818 raft_consensus.cc:348] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.003293 16818 raft_consensus.cc:374] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:16.003623 16818 raft_consensus.cc:729] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:16.003741 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:16.004223 16818 consensus_queue.cc:260] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.005865 16573 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:16.013397 16818 ts_tablet_manager.cc:1428] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.029s user 0.003s sys 0.004s
I20260810 18:51:16.014381 16818 tablet_bootstrap.cc:492] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:16.020294 17475 raft_consensus.cc:2726] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:16.026165 16818 tablet_bootstrap.cc:654] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:16.034032 17384 consensus_queue.cc:1035] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:16.040094 16818 tablet_bootstrap.cc:492] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:16.040578 16818 ts_tablet_manager.cc:1397] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.010s
I20260810 18:51:16.042886 16818 raft_consensus.cc:348] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.043589 16818 raft_consensus.cc:374] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:16.043843 16818 raft_consensus.cc:729] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:16.044468 16818 consensus_queue.cc:260] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.046841 16818 ts_tablet_manager.cc:1428] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:51:16.047804 16818 tablet_bootstrap.cc:492] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:16.054338 16818 tablet_bootstrap.cc:654] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
W20260810 18:51:16.059511 16448 leader_election.cc:341] T abdb4e808ce3454ab9a52051b369e5d0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:16.060302 16448 leader_election.cc:302] T abdb4e808ce3454ab9a52051b369e5d0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:16.061269 17457 raft_consensus.cc:2726] T abdb4e808ce3454ab9a52051b369e5d0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:16.062781 17457 consensus_queue.cc:1035] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:51:16.066526 16958 raft_consensus.cc:482] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:16.066994 16958 raft_consensus.cc:504] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.068751 16958 leader_election.cc:288] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:16.073869 17350 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:16.074473 17350 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.077692 17350 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:16.079121 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:16.082001 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:16.083853 16701 leader_election.cc:341] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:16.088438 16817 tablet_bootstrap.cc:654] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
W20260810 18:51:16.091806 16448 consensus_peers.cc:487] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea -> Peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Couldn't send request to peer 0397d760589d40d3b92ef141cdfc2dbb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:16.095556 16504 raft_consensus.cc:3037] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:16.101575 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:16.101737 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:16.102316 16508 raft_consensus.cc:2454] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 0.
I20260810 18:51:16.112493 16695 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:16.113677 16695 leader_election.cc:302] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:16.114892 17349 raft_consensus.cc:2726] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:16.114979 17350 raft_consensus.cc:2781] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:16.119464 17350 raft_consensus.cc:482] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:16.119791 17350 raft_consensus.cc:3037] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:16.123346 17450 consensus_queue.cc:1035] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:16.126171 17475 consensus_queue.cc:1035] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:51:16.154007 17451 raft_consensus.cc:482] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:16.154524 17451 raft_consensus.cc:504] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.156561 17451 leader_election.cc:288] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:16.157346 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:16.159091 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:16.159194 17350 raft_consensus.cc:504] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.163856 16450 leader_election.cc:302] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:16.165385 17451 raft_consensus.cc:2726] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:16.172461 16814 tablet_bootstrap.cc:492] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:16.172972 16814 ts_tablet_manager.cc:1397] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.238s user 0.029s sys 0.047s
I20260810 18:51:16.174451 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:51:16.176432 16701 leader_election.cc:341] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:16.186710 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:16.187368 16507 raft_consensus.cc:3037] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:16.173638 17350 leader_election.cc:288] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:16.210471 17475 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:16.221966 16814 raft_consensus.cc:348] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.222857 16814 raft_consensus.cc:374] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:16.223166 16814 raft_consensus.cc:729] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:16.223351 16818 tablet_bootstrap.cc:492] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:16.223850 16818 ts_tablet_manager.cc:1397] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.176s user 0.042s sys 0.057s
I20260810 18:51:16.223783 16814 consensus_queue.cc:260] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.222622 17475 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.227362 17475 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:16.229915 16507 raft_consensus.cc:2454] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:16.231226 16695 leader_election.cc:302] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:16.231892 16958 raft_consensus.cc:2781] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:16.232312 16958 raft_consensus.cc:686] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:16.235472 16817 tablet_bootstrap.cc:492] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:16.235667 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:16.235921 16817 ts_tablet_manager.cc:1397] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.252s user 0.010s sys 0.010s
I20260810 18:51:16.232956 16958 consensus_queue.cc:237] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.236128 16818 raft_consensus.cc:348] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.237600 16818 raft_consensus.cc:374] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:16.238814 16818 raft_consensus.cc:729] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:16.239567 16818 consensus_queue.cc:260] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.241211 16818 ts_tablet_manager.cc:1428] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260810 18:51:16.242372 16818 tablet_bootstrap.cc:492] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:16.245189 16817 raft_consensus.cc:348] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.245679 16817 raft_consensus.cc:374] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:16.245975 16817 raft_consensus.cc:729] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:16.246483 16817 consensus_queue.cc:260] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:16.248906 16575 leader_election.cc:341] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:16.251015 17451 consensus_queue.cc:1035] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:51:16.251730 16814 ts_tablet_manager.cc:1428] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.078s user 0.021s sys 0.024s
I20260810 18:51:16.252656 16814 tablet_bootstrap.cc:492] T 649d43610ba24875a4418e39b25126ba P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:16.264232 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:16.265661 17372 raft_consensus.cc:482] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:16.266139 16817 ts_tablet_manager.cc:1428] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.030s user 0.004s sys 0.002s
I20260810 18:51:16.267153 16747 raft_consensus.cc:3037] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
W20260810 18:51:16.270546 16573 leader_election.cc:341] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:16.271000 16573 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:16.266106 17372 raft_consensus.cc:504] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.271915 17475 raft_consensus.cc:2726] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:16.274116 17372 leader_election.cc:288] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:16.278877 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:16.282083 16448 leader_election.cc:341] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:16.289867 16817 tablet_bootstrap.cc:492] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:16.290799 16814 tablet_bootstrap.cc:654] T 649d43610ba24875a4418e39b25126ba P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:16.306330 17384 raft_consensus.cc:482] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:16.325037 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:51:16.327378 16450 leader_election.cc:341] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:16.327873 16450 leader_election.cc:302] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:16.330915 17451 raft_consensus.cc:2726] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:16.318778 17384 raft_consensus.cc:504] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.339838 17384 leader_election.cc:288] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:16.343721 16818 tablet_bootstrap.cc:654] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:16.350597 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:16.351125 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:16.363606 16575 leader_election.cc:302] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:16.369637 17386 raft_consensus.cc:2726] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:16.376521 16817 tablet_bootstrap.cc:654] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:16.411960 16344 catalog_manager.cc:5495] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 0 to 1, leader changed from <none> to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:51:16.452667 16818 tablet_bootstrap.cc:492] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:16.455520 16818 ts_tablet_manager.cc:1397] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.213s user 0.015s sys 0.019s
I20260810 18:51:16.461864 16814 tablet_bootstrap.cc:492] T 649d43610ba24875a4418e39b25126ba P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:16.462487 16814 ts_tablet_manager.cc:1397] T 649d43610ba24875a4418e39b25126ba P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.210s user 0.017s sys 0.011s
I20260810 18:51:16.465775 16814 raft_consensus.cc:348] T 649d43610ba24875a4418e39b25126ba P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.466297 16814 raft_consensus.cc:374] T 649d43610ba24875a4418e39b25126ba P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:16.466527 16814 raft_consensus.cc:729] T 649d43610ba24875a4418e39b25126ba P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:16.467159 16814 consensus_queue.cc:260] T 649d43610ba24875a4418e39b25126ba P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.470089 16814 ts_tablet_manager.cc:1428] T 649d43610ba24875a4418e39b25126ba P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20260810 18:51:16.470837 16818 raft_consensus.cc:348] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.471592 16818 raft_consensus.cc:374] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:16.471853 16818 raft_consensus.cc:729] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:16.473719 16818 consensus_queue.cc:260] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.476377 16814 tablet_bootstrap.cc:492] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:16.478226 16817 tablet_bootstrap.cc:492] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:16.478698 16817 ts_tablet_manager.cc:1397] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.189s user 0.018s sys 0.006s
I20260810 18:51:16.481936 16818 ts_tablet_manager.cc:1428] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s
I20260810 18:51:16.482062 16817 raft_consensus.cc:348] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.482736 16817 raft_consensus.cc:374] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:16.483008 16818 tablet_bootstrap.cc:492] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:16.482987 16817 raft_consensus.cc:729] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:16.484282 16817 consensus_queue.cc:260] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.488313 16814 tablet_bootstrap.cc:654] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:16.491981 16817 ts_tablet_manager.cc:1428] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20260810 18:51:16.492226 16636 raft_consensus.cc:3037] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:16.492918 16817 tablet_bootstrap.cc:492] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:16.497140 16818 tablet_bootstrap.cc:654] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:16.498399 16958 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:16.499917 16958 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.505089 16958 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:16.505869 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:16.512234 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:16.513139 16633 raft_consensus.cc:2454] T 649d43610ba24875a4418e39b25126ba P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 0.
I20260810 18:51:16.515359 16701 leader_election.cc:302] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:16.516244 16958 raft_consensus.cc:2781] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:16.516848 16958 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:16.517160 16958 raft_consensus.cc:3037] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:16.515906 16504 raft_consensus.cc:2454] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 0.
I20260810 18:51:16.531112 16958 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.533319 16958 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:16.534312 16817 tablet_bootstrap.cc:654] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:16.534296 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:16.534296 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:16.535301 16633 raft_consensus.cc:3037] T 649d43610ba24875a4418e39b25126ba P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:16.535264 16504 raft_consensus.cc:3037] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:16.572418 16504 raft_consensus.cc:2454] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:16.576943 16633 raft_consensus.cc:2454] T 649d43610ba24875a4418e39b25126ba P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:16.586440 16701 leader_election.cc:302] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:16.587267 16958 raft_consensus.cc:2781] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:16.587702 16958 raft_consensus.cc:686] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:16.588459 16958 consensus_queue.cc:237] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.578647 16818 tablet_bootstrap.cc:492] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:16.590471 16818 ts_tablet_manager.cc:1397] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.108s user 0.012s sys 0.024s
I20260810 18:51:16.593060 16818 raft_consensus.cc:348] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.593681 16818 raft_consensus.cc:374] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:16.593937 16818 raft_consensus.cc:729] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:16.594523 16818 consensus_queue.cc:260] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.596205 16818 ts_tablet_manager.cc:1428] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260810 18:51:16.597468 16818 tablet_bootstrap.cc:492] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:16.602943 16818 tablet_bootstrap.cc:654] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:16.581908 16814 tablet_bootstrap.cc:492] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:16.585266 16817 tablet_bootstrap.cc:492] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:16.616873 16817 ts_tablet_manager.cc:1397] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.124s user 0.006s sys 0.011s
I20260810 18:51:16.619665 16817 raft_consensus.cc:348] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.620404 16817 raft_consensus.cc:374] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:16.620658 16817 raft_consensus.cc:729] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:16.621286 16817 consensus_queue.cc:260] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.623339 16814 ts_tablet_manager.cc:1397] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.147s user 0.016s sys 0.020s
I20260810 18:51:16.625963 16814 raft_consensus.cc:348] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.626673 16814 raft_consensus.cc:374] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:16.629812 16817 ts_tablet_manager.cc:1428] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260810 18:51:16.625316 16344 catalog_manager.cc:5495] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 0 to 1, leader changed from <none> to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:51:16.630918 16817 tablet_bootstrap.cc:492] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:16.626926 16814 raft_consensus.cc:729] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:16.637816 16814 consensus_queue.cc:260] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.640358 16814 ts_tablet_manager.cc:1428] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.017s user 0.005s sys 0.001s
I20260810 18:51:16.641323 16814 tablet_bootstrap.cc:492] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:16.649330 16958 consensus_queue.cc:1035] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:51:16.661623 16817 tablet_bootstrap.cc:654] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:16.662724 16818 tablet_bootstrap.cc:492] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:16.663236 16818 ts_tablet_manager.cc:1397] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.066s user 0.037s sys 0.027s
I20260810 18:51:16.667887 16818 raft_consensus.cc:348] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.668572 16818 raft_consensus.cc:374] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:16.668828 16818 raft_consensus.cc:729] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:16.669459 16818 consensus_queue.cc:260] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.671710 16818 ts_tablet_manager.cc:1428] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260810 18:51:16.672652 16818 tablet_bootstrap.cc:492] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:16.675138 16817 tablet_bootstrap.cc:492] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:16.675647 16817 ts_tablet_manager.cc:1397] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.045s user 0.012s sys 0.000s
I20260810 18:51:16.676744 16814 tablet_bootstrap.cc:654] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:16.678023 16817 raft_consensus.cc:348] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.678753 16818 tablet_bootstrap.cc:654] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:16.678753 16817 raft_consensus.cc:374] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:16.679222 16817 raft_consensus.cc:729] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:16.679838 16817 consensus_queue.cc:260] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.682013 16817 ts_tablet_manager.cc:1428] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:51:16.683931 17362 raft_consensus.cc:482] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:16.684373 17362 raft_consensus.cc:504] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.686259 17362 leader_election.cc:288] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:16.688241 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:16.688603 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:16.689050 16750 raft_consensus.cc:2454] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 0.
W20260810 18:51:16.690212 16448 leader_election.cc:341] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:16.690941 16450 leader_election.cc:302] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
W20260810 18:51:16.691758 16701 consensus_peers.cc:487] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be -> Peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Couldn't send request to peer 0397d760589d40d3b92ef141cdfc2dbb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:16.692070 17362 raft_consensus.cc:2781] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:16.692926 17362 raft_consensus.cc:482] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:16.693241 17362 raft_consensus.cc:3037] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:16.701628 17362 raft_consensus.cc:504] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.705060 16817 tablet_bootstrap.cc:492] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:16.737766 16817 tablet_bootstrap.cc:654] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:16.723191 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:16.719413 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:16.710988 16814 tablet_bootstrap.cc:492] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:16.761804 16747 raft_consensus.cc:3037] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:16.762108 16814 ts_tablet_manager.cc:1397] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.121s user 0.010s sys 0.007s
I20260810 18:51:16.762970 16818 tablet_bootstrap.cc:492] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:16.763460 16818 ts_tablet_manager.cc:1397] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.091s user 0.028s sys 0.039s
I20260810 18:51:16.766939 16818 raft_consensus.cc:348] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.767711 16818 raft_consensus.cc:374] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:16.767987 16818 raft_consensus.cc:729] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:16.768586 16818 consensus_queue.cc:260] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.719259 17452 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:16.770095 16747 raft_consensus.cc:2454] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:16.769479 17452 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.771029 16818 ts_tablet_manager.cc:1428] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260810 18:51:16.774438 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:16.775100 16818 tablet_bootstrap.cc:492] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:16.776295 16814 raft_consensus.cc:348] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.717183 17362 leader_election.cc:288] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:16.776978 16814 raft_consensus.cc:374] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:16.777302 16814 raft_consensus.cc:729] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:16.771665 16450 leader_election.cc:302] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:51:16.778162 16448 leader_election.cc:341] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:16.778088 16814 consensus_queue.cc:260] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.779999 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:16.781111 16750 raft_consensus.cc:2454] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 0.
I20260810 18:51:16.781491 16814 ts_tablet_manager.cc:1428] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260810 18:51:16.782857 16450 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:16.783167 16814 tablet_bootstrap.cc:492] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:16.783859 17372 raft_consensus.cc:2781] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:16.784528 17372 raft_consensus.cc:686] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:16.786173 17450 raft_consensus.cc:2781] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:16.785640 17372 consensus_queue.cc:237] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.787218 17450 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:16.772989 17452 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:16.788326 17333 raft_consensus.cc:482] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:16.788890 17333 raft_consensus.cc:504] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.787755 17450 raft_consensus.cc:3037] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
W20260810 18:51:16.793342 16448 leader_election.cc:341] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:16.797060 16817 tablet_bootstrap.cc:492] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:16.797511 16817 ts_tablet_manager.cc:1397] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.093s user 0.019s sys 0.012s
I20260810 18:51:16.798355 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:16.799191 16504 raft_consensus.cc:2454] T abdb4e808ce3454ab9a52051b369e5d0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 0.
I20260810 18:51:16.800011 16818 tablet_bootstrap.cc:654] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:16.801707 17333 leader_election.cc:288] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:16.803972 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:16.805625 16573 leader_election.cc:302] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:16.808878 17333 raft_consensus.cc:2781] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:16.810175 17333 raft_consensus.cc:482] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:16.832003 17333 raft_consensus.cc:3037] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
W20260810 18:51:16.848227 16575 leader_election.cc:341] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:16.860430 17452 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:16.867841 17452 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.861690 17450 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.869176 16817 raft_consensus.cc:348] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.870360 16817 raft_consensus.cc:374] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:16.870622 16817 raft_consensus.cc:729] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:16.871387 16817 consensus_queue.cc:260] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.874228 16817 ts_tablet_manager.cc:1428] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.076s user 0.007s sys 0.000s
I20260810 18:51:16.879907 16344 catalog_manager.cc:5495] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 0 to 1, leader changed from <none> to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:16.883297 16818 tablet_bootstrap.cc:492] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:16.883759 16818 ts_tablet_manager.cc:1397] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.109s user 0.035s sys 0.046s
I20260810 18:51:16.889051 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:51:16.890307 16448 leader_election.cc:341] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:16.894011 17452 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:16.880872 17450 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:16.896133 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:16.898716 16448 leader_election.cc:341] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:16.904296 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:16.904912 16749 raft_consensus.cc:3037] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:16.905385 16817 tablet_bootstrap.cc:492] T 3f3efea4006f43ee933f50d3e328a1f0 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:16.906107 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:16.906669 16750 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 0.
I20260810 18:51:16.908222 17333 raft_consensus.cc:504] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.910491 17333 leader_election.cc:288] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:16.911852 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:16.912526 16508 raft_consensus.cc:3037] T abdb4e808ce3454ab9a52051b369e5d0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:16.915123 16450 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:16.915859 16749 raft_consensus.cc:2454] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:16.916693 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:16.918526 16750 raft_consensus.cc:3037] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:16.919315 16450 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:16.920292 17362 raft_consensus.cc:2781] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:16.920852 17362 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:16.921255 17362 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:16.921780 17450 raft_consensus.cc:2781] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:16.922427 17450 raft_consensus.cc:686] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:16.923524 17450 consensus_queue.cc:237] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.917681 16818 raft_consensus.cc:348] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.925967 16818 raft_consensus.cc:374] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:16.926244 16818 raft_consensus.cc:729] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:16.927083 16818 consensus_queue.cc:260] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.930243 16750 raft_consensus.cc:2454] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:16.932149 16575 leader_election.cc:302] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:16.933920 17386 raft_consensus.cc:2781] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:16.934754 17386 raft_consensus.cc:686] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:16.936363 17386 consensus_queue.cc:237] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.944926 16814 tablet_bootstrap.cc:654] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:16.950172 16818 ts_tablet_manager.cc:1428] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.066s user 0.006s sys 0.003s
I20260810 18:51:16.956585 16818 tablet_bootstrap.cc:492] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:16.960421 17362 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:16.964157 16344 catalog_manager.cc:5495] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 0 to 1, leader changed from <none> to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:16.970914 16345 catalog_manager.cc:5495] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:16.988821 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:16.989357 16508 raft_consensus.cc:2454] T abdb4e808ce3454ab9a52051b369e5d0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
W20260810 18:51:16.990672 16448 leader_election.cc:341] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:16.991664 17362 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:16.992283 16818 tablet_bootstrap.cc:654] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:16.993208 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:16.993851 16750 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:16.997030 16817 tablet_bootstrap.cc:654] T 3f3efea4006f43ee933f50d3e328a1f0 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:17.002521 16814 tablet_bootstrap.cc:492] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:17.003398 16814 ts_tablet_manager.cc:1397] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.220s user 0.014s sys 0.018s
I20260810 18:51:17.007210 16750 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:17.009074 16450 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:17.009896 17025 raft_consensus.cc:482] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:17.010591 17025 raft_consensus.cc:504] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.011273 17450 raft_consensus.cc:2781] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:17.011936 17450 raft_consensus.cc:686] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:17.013579 17025 leader_election.cc:288] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:17.013336 17450 consensus_queue.cc:237] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.014683 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:17.015924 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:17.016345 16626 raft_consensus.cc:2454] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 0.
I20260810 18:51:17.016644 16507 raft_consensus.cc:2454] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 0.
I20260810 18:51:17.017987 16701 leader_election.cc:302] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:17.019568 17025 raft_consensus.cc:2781] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:17.019958 17025 raft_consensus.cc:482] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:17.020323 17025 raft_consensus.cc:3037] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:17.020879 16814 raft_consensus.cc:348] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.021735 16814 raft_consensus.cc:374] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:17.022085 16814 raft_consensus.cc:729] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:17.038825 16814 consensus_queue.cc:260] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.041355 16814 ts_tablet_manager.cc:1428] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.037s user 0.004s sys 0.004s
I20260810 18:51:17.042461 16814 tablet_bootstrap.cc:492] T 694fa0ce1cd246a985705792780adee0 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:17.042810 16344 catalog_manager.cc:5495] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 0 to 1, leader changed from <none> to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:17.046008 17025 raft_consensus.cc:504] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.059345 16817 tablet_bootstrap.cc:492] T 3f3efea4006f43ee933f50d3e328a1f0 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:17.060894 16817 ts_tablet_manager.cc:1397] T 3f3efea4006f43ee933f50d3e328a1f0 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.175s user 0.019s sys 0.007s
I20260810 18:51:17.060279 16814 tablet_bootstrap.cc:654] T 694fa0ce1cd246a985705792780adee0 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:17.065667 17025 leader_election.cc:288] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:17.066819 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:17.066797 16817 raft_consensus.cc:348] T 3f3efea4006f43ee933f50d3e328a1f0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.072304 16626 raft_consensus.cc:3037] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:17.072645 16817 raft_consensus.cc:374] T 3f3efea4006f43ee933f50d3e328a1f0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:17.073108 16817 raft_consensus.cc:729] T 3f3efea4006f43ee933f50d3e328a1f0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:17.073948 16817 consensus_queue.cc:260] T 3f3efea4006f43ee933f50d3e328a1f0 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.076973 16817 ts_tablet_manager.cc:1428] T 3f3efea4006f43ee933f50d3e328a1f0 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.016s user 0.008s sys 0.001s
I20260810 18:51:17.069461 16818 tablet_bootstrap.cc:492] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:17.078770 16818 ts_tablet_manager.cc:1397] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.122s user 0.013s sys 0.005s
I20260810 18:51:17.079629 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:17.080710 16508 raft_consensus.cc:3037] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:17.084954 16817 tablet_bootstrap.cc:492] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:17.093595 16508 raft_consensus.cc:2454] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:17.105849 16695 leader_election.cc:302] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:17.105715 17452 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:17.106623 17452 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.108530 17025 raft_consensus.cc:2781] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:17.109007 17025 raft_consensus.cc:686] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:17.110358 16818 raft_consensus.cc:348] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.111056 16818 raft_consensus.cc:374] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:17.111385 16818 raft_consensus.cc:729] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:17.113853 17452 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:17.116994 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:17.109730 17025 consensus_queue.cc:237] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.118471 16633 raft_consensus.cc:2454] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 0.
I20260810 18:51:17.112007 16818 consensus_queue.cc:260] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.120540 16448 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:17.122304 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:17.122733 16626 raft_consensus.cc:2454] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
W20260810 18:51:17.124403 16450 leader_election.cc:341] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:17.127879 17452 raft_consensus.cc:2781] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:17.128259 17452 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:17.128564 17452 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:17.128968 16818 ts_tablet_manager.cc:1428] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.050s user 0.008s sys 0.000s
I20260810 18:51:17.130239 16818 tablet_bootstrap.cc:492] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
W20260810 18:51:17.133620 16415 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260810 18:51:17.135718 17349 consensus_queue.cc:1035] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:51:17.136734 17362 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:17.137228 17362 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.139490 17362 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:17.144232 16817 tablet_bootstrap.cc:654] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:17.145246 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:17.145749 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:17.150276 16818 tablet_bootstrap.cc:654] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:17.152401 17452 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.154147 16344 catalog_manager.cc:5495] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 0 to 1, leader changed from <none> to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:51:17.155380 17452 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:51:17.157696 16448 leader_election.cc:341] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:17.157980 16814 tablet_bootstrap.cc:492] T 694fa0ce1cd246a985705792780adee0 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:17.161284 16814 ts_tablet_manager.cc:1397] T 694fa0ce1cd246a985705792780adee0 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.119s user 0.015s sys 0.003s
I20260810 18:51:17.158907 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:51:17.162042 16450 leader_election.cc:341] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:17.162159 16633 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:17.164906 16814 raft_consensus.cc:348] T 694fa0ce1cd246a985705792780adee0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.165686 16814 raft_consensus.cc:374] T 694fa0ce1cd246a985705792780adee0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:17.166060 16814 raft_consensus.cc:729] T 694fa0ce1cd246a985705792780adee0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:17.165943 16450 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:17.167513 17372 raft_consensus.cc:2726] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:17.169260 16817 tablet_bootstrap.cc:492] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:17.169684 16817 ts_tablet_manager.cc:1397] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.091s user 0.013s sys 0.001s
I20260810 18:51:17.170919 16814 consensus_queue.cc:260] T 694fa0ce1cd246a985705792780adee0 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.173063 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
W20260810 18:51:17.174401 16450 leader_election.cc:341] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:17.176095 16817 raft_consensus.cc:348] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.176764 16817 raft_consensus.cc:374] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:17.177023 16817 raft_consensus.cc:729] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:17.177609 16817 consensus_queue.cc:260] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.179723 16817 ts_tablet_manager.cc:1428] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260810 18:51:17.181274 16817 tablet_bootstrap.cc:492] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:17.182638 16814 ts_tablet_manager.cc:1428] T 694fa0ce1cd246a985705792780adee0 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.021s user 0.003s sys 0.003s
I20260810 18:51:17.183542 16814 tablet_bootstrap.cc:492] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:17.184443 16818 tablet_bootstrap.cc:492] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:17.184850 16818 ts_tablet_manager.cc:1397] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.055s user 0.008s sys 0.003s
I20260810 18:51:17.187858 17025 consensus_queue.cc:1035] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:51:17.188231 16818 raft_consensus.cc:348] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.188951 16818 raft_consensus.cc:374] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:17.189204 16818 raft_consensus.cc:729] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:17.189332 16633 raft_consensus.cc:2454] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:17.189769 16818 consensus_queue.cc:260] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.191382 16818 ts_tablet_manager.cc:1428] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260810 18:51:17.192291 16818 tablet_bootstrap.cc:492] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea: Bootstrap starting.
I20260810 18:51:17.197328 17333 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:17.197805 17333 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.198920 16818 tablet_bootstrap.cc:654] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:17.199885 17333 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:17.201931 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:17.202554 16750 raft_consensus.cc:2454] T 3f3efea4006f43ee933f50d3e328a1f0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 0.
I20260810 18:51:17.204458 16448 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:17.206151 16575 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:17.206902 17333 raft_consensus.cc:2781] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:17.207255 17333 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:17.207394 17569 raft_consensus.cc:2781] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:17.207609 17333 raft_consensus.cc:3037] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:17.207924 17569 raft_consensus.cc:686] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:17.208503 16817 tablet_bootstrap.cc:654] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:17.208755 17569 consensus_queue.cc:237] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.203502 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:51:17.217422 16573 leader_election.cc:341] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:17.218289 16814 tablet_bootstrap.cc:654] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:17.248894 16344 catalog_manager.cc:5495] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 0 to 1, leader changed from <none> to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:17.300730 17333 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.326978 17333 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:17.328217 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:17.329572 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:17.330464 16747 raft_consensus.cc:3037] T 3f3efea4006f43ee933f50d3e328a1f0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:17.331905 16818 tablet_bootstrap.cc:492] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea: No bootstrap required, opened a new log
I20260810 18:51:17.332401 16818 ts_tablet_manager.cc:1397] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea: Time spent bootstrapping tablet: real 0.140s user 0.021s sys 0.020s
W20260810 18:51:17.334523 16573 leader_election.cc:341] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:17.337103 16814 tablet_bootstrap.cc:492] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:17.340040 16814 ts_tablet_manager.cc:1397] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.157s user 0.016s sys 0.013s
I20260810 18:51:17.343349 16814 raft_consensus.cc:348] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.344030 16814 raft_consensus.cc:374] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:17.344314 16814 raft_consensus.cc:729] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:17.345042 16814 consensus_queue.cc:260] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.340893 16817 tablet_bootstrap.cc:492] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:17.346946 16817 ts_tablet_manager.cc:1397] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.166s user 0.043s sys 0.029s
I20260810 18:51:17.349992 16817 raft_consensus.cc:348] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.350657 16817 raft_consensus.cc:374] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:17.350911 16817 raft_consensus.cc:729] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:17.351596 16817 consensus_queue.cc:260] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.348438 16747 raft_consensus.cc:2454] T 3f3efea4006f43ee933f50d3e328a1f0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:17.354043 17450 consensus_queue.cc:1035] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:51:17.354841 16818 raft_consensus.cc:348] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.355774 16818 raft_consensus.cc:374] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:17.355849 16575 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:17.353612 16814 ts_tablet_manager.cc:1428] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.013s user 0.005s sys 0.002s
I20260810 18:51:17.356894 17481 raft_consensus.cc:2781] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:17.357239 16814 tablet_bootstrap.cc:492] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:17.357368 17481 raft_consensus.cc:686] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:17.356160 16818 raft_consensus.cc:729] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Initialized, Role: FOLLOWER
I20260810 18:51:17.361824 16818 consensus_queue.cc:260] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.362941 17372 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:17.363381 17372 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.364802 16818 ts_tablet_manager.cc:1428] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea: Time spent starting tablet: real 0.032s user 0.009s sys 0.000s
I20260810 18:51:17.358248 17481 consensus_queue.cc:237] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.361147 16817 ts_tablet_manager.cc:1428] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20260810 18:51:17.365289 16814 tablet_bootstrap.cc:654] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:17.365311 17372 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:17.367128 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:17.369194 16448 consensus_peers.cc:487] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea -> Peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Couldn't send request to peer 0397d760589d40d3b92ef141cdfc2dbb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:17.366648 16817 tablet_bootstrap.cc:492] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:17.368125 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:17.378129 16750 raft_consensus.cc:2379] T 3f3efea4006f43ee933f50d3e328a1f0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 1: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:17.379142 16450 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:17.379942 17372 raft_consensus.cc:3037] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:17.380517 16814 tablet_bootstrap.cc:492] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:17.381019 16814 ts_tablet_manager.cc:1397] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.001s
I20260810 18:51:17.383431 16814 raft_consensus.cc:348] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.384074 16814 raft_consensus.cc:374] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:17.384305 16814 raft_consensus.cc:729] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:17.384876 16814 consensus_queue.cc:260] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.389530 17372 raft_consensus.cc:2726] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:17.387006 16814 ts_tablet_manager.cc:1428] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260810 18:51:17.390828 16814 tablet_bootstrap.cc:492] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:17.395051 16817 tablet_bootstrap.cc:654] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:17.402148 17333 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:17.402992 17333 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.406733 16814 tablet_bootstrap.cc:654] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:17.406502 16345 catalog_manager.cc:5495] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:17.413583 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:17.417178 17362 consensus_queue.cc:1035] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:17.432107 16633 raft_consensus.cc:3037] T 694fa0ce1cd246a985705792780adee0 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:17.434640 17333 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:17.439733 16814 tablet_bootstrap.cc:492] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:17.440237 16814 ts_tablet_manager.cc:1397] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.050s user 0.023s sys 0.008s
I20260810 18:51:17.445704 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:17.446552 16750 raft_consensus.cc:2379] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 1: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:17.452037 17349 raft_consensus.cc:482] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:17.452723 17349 raft_consensus.cc:504] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.458228 16814 raft_consensus.cc:348] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.462778 16817 tablet_bootstrap.cc:492] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:17.463301 16817 ts_tablet_manager.cc:1397] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.097s user 0.026s sys 0.018s
I20260810 18:51:17.462821 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:17.459270 16814 raft_consensus.cc:374] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:17.464835 16814 raft_consensus.cc:729] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:17.465860 16814 consensus_queue.cc:260] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.466527 16817 raft_consensus.cc:348] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.467882 17362 consensus_queue.cc:1035] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:17.468597 16817 raft_consensus.cc:374] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:17.464809 16508 raft_consensus.cc:2454] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 0.
I20260810 18:51:17.468967 16817 raft_consensus.cc:729] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:17.469779 16817 consensus_queue.cc:260] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.472306 16817 ts_tablet_manager.cc:1428] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.009s user 0.001s sys 0.006s
I20260810 18:51:17.473441 16817 tablet_bootstrap.cc:492] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be: Bootstrap starting.
I20260810 18:51:17.473337 16695 leader_election.cc:302] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:17.474992 17025 raft_consensus.cc:2781] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:17.475361 17025 raft_consensus.cc:482] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:17.475728 17025 raft_consensus.cc:3037] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:17.478034 16814 ts_tablet_manager.cc:1428] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.037s user 0.009s sys 0.000s
I20260810 18:51:17.479172 16814 tablet_bootstrap.cc:492] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:17.480317 16817 tablet_bootstrap.cc:654] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:17.480983 16958 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:17.481468 16958 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.484052 16958 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:17.486496 16814 tablet_bootstrap.cc:654] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:17.487907 16626 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:17.456863 17349 leader_election.cc:288] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:17.517714 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:17.526499 16814 tablet_bootstrap.cc:492] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:17.527212 16814 ts_tablet_manager.cc:1397] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.048s user 0.025s sys 0.003s
I20260810 18:51:17.527973 16575 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:17.534212 17569 consensus_queue.cc:1035] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:17.534719 16814 raft_consensus.cc:348] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.535650 16814 raft_consensus.cc:374] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:17.536113 16814 raft_consensus.cc:729] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:17.536906 16814 consensus_queue.cc:260] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.526736 17025 raft_consensus.cc:504] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.544544 16814 ts_tablet_manager.cc:1428] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260810 18:51:17.544931 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:17.545866 16814 tablet_bootstrap.cc:492] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:17.545293 17452 consensus_queue.cc:1035] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:17.546058 16623 raft_consensus.cc:2379] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 1: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:17.553956 16701 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:17.555409 16958 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:17.558252 17025 leader_election.cc:288] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:17.562268 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:17.562839 16505 raft_consensus.cc:3037] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
W20260810 18:51:17.641217 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 69157.3us
I20260810 18:51:17.641878 17481 consensus_queue.cc:1035] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:17.640547 17475 raft_consensus.cc:2726] T 694fa0ce1cd246a985705792780adee0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:17.647161 16505 raft_consensus.cc:2454] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:17.657497 16958 raft_consensus.cc:2726] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:17.639245 17333 raft_consensus.cc:482] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:17.668413 16814 tablet_bootstrap.cc:654] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:17.641106 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:17.647850 17481 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:17.679662 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:51:17.753481 16701 leader_election.cc:341] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:17.668416 17333 raft_consensus.cc:504] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.674834 17481 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.789970 16817 tablet_bootstrap.cc:492] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be: No bootstrap required, opened a new log
I20260810 18:51:17.790484 16817 ts_tablet_manager.cc:1397] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent bootstrapping tablet: real 0.317s user 0.025s sys 0.031s
I20260810 18:51:17.888973 16817 raft_consensus.cc:348] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.889725 16817 raft_consensus.cc:374] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:17.889995 16817 raft_consensus.cc:729] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Initialized, Role: FOLLOWER
I20260810 18:51:17.891839 17025 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:17.892400 17025 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.892823 17333 leader_election.cc:288] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:17.895973 17025 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:17.908881 17481 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:17.890707 16817 consensus_queue.cc:260] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.921155 16695 leader_election.cc:302] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:17.928589 16958 raft_consensus.cc:2781] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:17.929239 16958 raft_consensus.cc:686] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:17.929796 16817 ts_tablet_manager.cc:1428] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent starting tablet: real 0.139s user 0.005s sys 0.004s
I20260810 18:51:17.930256 16958 consensus_queue.cc:237] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.949771 17569 raft_consensus.cc:482] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:17.950305 17569 raft_consensus.cc:504] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:17.952309 17569 leader_election.cc:288] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:17.956938 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:17.957602 16507 raft_consensus.cc:2454] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 0.
I20260810 18:51:17.960017 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:17.960645 16506 raft_consensus.cc:2454] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 0.
I20260810 18:51:17.968191 16573 leader_election.cc:302] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:17.969205 16573 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:17.971585 17620 raft_consensus.cc:2781] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:17.980667 17620 raft_consensus.cc:482] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:17.983269 17386 raft_consensus.cc:2781] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:18.001850 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:18.005406 16814 tablet_bootstrap.cc:492] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:18.005950 16814 ts_tablet_manager.cc:1397] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.460s user 0.018s sys 0.034s
I20260810 18:51:17.999042 17620 raft_consensus.cc:3037] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:17.999114 17386 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:17.998119 17630 raft_consensus.cc:482] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:18.017946 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:18.018812 16505 raft_consensus.cc:2454] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 0.
I20260810 18:51:18.021276 16814 raft_consensus.cc:348] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:18.022131 16814 raft_consensus.cc:374] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:18.022470 16814 raft_consensus.cc:729] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:18.015873 17386 raft_consensus.cc:3037] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:18.017055 17630 raft_consensus.cc:504] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:18.029160 16814 consensus_queue.cc:260] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:18.032852 16814 ts_tablet_manager.cc:1428] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.027s user 0.008s sys 0.000s
W20260810 18:51:18.040558 16701 leader_election.cc:341] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:51:18.042526 16814 tablet_bootstrap.cc:492] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb: Bootstrap starting.
I20260810 18:51:18.045137 17450 consensus_queue.cc:1035] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:18.033031 17630 leader_election.cc:288] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:18.061671 17630 consensus_queue.cc:1035] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:18.067369 17627 raft_consensus.cc:482] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:18.068179 17627 raft_consensus.cc:504] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:18.068374 16338 catalog_manager.cc:5495] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 0 to 1, leader changed from <none> to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:51:18.073312 17627 leader_election.cc:288] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:18.080560 17386 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:18.100571 17386 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:18.102138 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:18.103037 16508 raft_consensus.cc:2454] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 0.
I20260810 18:51:18.103855 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:18.104700 16746 raft_consensus.cc:2454] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 0.
I20260810 18:51:18.118170 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:18.119422 16748 raft_consensus.cc:2454] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 0.
I20260810 18:51:18.123617 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:18.124470 16746 raft_consensus.cc:2454] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 0.
I20260810 18:51:18.142859 16814 tablet_bootstrap.cc:654] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb: Neither blocks nor log segments found. Creating new log.
I20260810 18:51:18.148012 16695 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:18.148679 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:18.119608 17620 raft_consensus.cc:504] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:18.162060 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:18.165419 17662 raft_consensus.cc:482] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:18.161865 17386 consensus_queue.cc:1035] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:18.169800 16636 raft_consensus.cc:2379] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 1: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:18.171185 17620 leader_election.cc:288] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:18.171554 17637 raft_consensus.cc:2781] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:18.172192 17637 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:18.152247 17633 consensus_queue.cc:1035] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.029s
I20260810 18:51:18.172797 17637 raft_consensus.cc:3037] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:18.166270 17662 raft_consensus.cc:504] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:18.144893 17635 consensus_queue.cc:1035] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.036s
I20260810 18:51:18.184791 16633 raft_consensus.cc:3037] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:18.197085 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:18.189214 16626 raft_consensus.cc:2379] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 1: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
W20260810 18:51:18.213637 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 58003.1us
I20260810 18:51:18.216042 17662 leader_election.cc:288] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:18.191412 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:18.204785 16508 raft_consensus.cc:3037] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:18.221073 16502 raft_consensus.cc:3037] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:18.235564 16450 leader_election.cc:302] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:18.248080 17386 consensus_queue.cc:1035] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:18.250944 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:18.269299 16695 leader_election.cc:302] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:18.275576 16958 raft_consensus.cc:2781] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:51:18.276773 16958 raft_consensus.cc:482] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:18.277089 16958 raft_consensus.cc:3037] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:18.274868 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:18.280022 16628 raft_consensus.cc:2379] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 1: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:18.281051 16814 tablet_bootstrap.cc:492] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb: No bootstrap required, opened a new log
I20260810 18:51:18.281571 16814 ts_tablet_manager.cc:1397] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb: Time spent bootstrapping tablet: real 0.239s user 0.022s sys 0.024s
I20260810 18:51:18.296139 16958 raft_consensus.cc:504] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:18.320152 16958 leader_election.cc:288] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:18.320566 16508 raft_consensus.cc:2454] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:18.321285 17450 raft_consensus.cc:2741] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260810 18:51:18.329440 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:18.330158 16504 raft_consensus.cc:2379] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 1: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
W20260810 18:51:18.340574 16415 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260810 18:51:18.344810 16958 consensus_queue.cc:1035] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 18:51:18.352233 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 66380.9us
I20260810 18:51:18.351995 17637 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:18.355734 16573 leader_election.cc:302] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:18.360147 16814 raft_consensus.cc:348] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:18.360872 16814 raft_consensus.cc:374] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:51:18.361140 16814 raft_consensus.cc:729] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Initialized, Role: FOLLOWER
I20260810 18:51:18.361749 16814 consensus_queue.cc:260] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:18.370272 17619 raft_consensus.cc:2781] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:51:18.375177 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:18.383654 17619 raft_consensus.cc:686] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:18.393006 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:18.394969 16814 ts_tablet_manager.cc:1428] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb: Time spent starting tablet: real 0.113s user 0.004s sys 0.003s
I20260810 18:51:18.397313 16575 leader_election.cc:302] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:18.398747 17386 raft_consensus.cc:2726] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:18.399617 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:18.400501 16750 raft_consensus.cc:2379] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 1: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:51:18.411497 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:18.417829 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:18.420676 16575 leader_election.cc:302] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:18.415584 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:18.422391 16743 raft_consensus.cc:2379] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 1: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:51:18.372430 17627 consensus_queue.cc:1035] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.052s
I20260810 18:51:18.427312 17475 raft_consensus.cc:2726] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:18.384745 17619 consensus_queue.cc:237] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:18.448330 17637 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:18.472937 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:18.473768 16632 raft_consensus.cc:2379] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 1: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:18.482656 17616 raft_consensus.cc:482] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:18.483125 17616 raft_consensus.cc:504] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:18.485057 17616 leader_election.cc:288] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:18.486435 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:18.497650 16624 raft_consensus.cc:3037] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
W20260810 18:51:18.523569 17678 log.cc:924] Time spent T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.068s user 0.000s sys 0.004s
I20260810 18:51:18.469981 16502 raft_consensus.cc:2454] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:18.531100 16701 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:18.533679 17025 raft_consensus.cc:2726] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 18:51:18.542044 16573 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:18.546955 17475 raft_consensus.cc:2781] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader election won for term 1
W20260810 18:51:18.551975 17687 log.cc:924] Time spent T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.148s user 0.002s sys 0.000s
I20260810 18:51:18.553998 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:18.562057 16622 raft_consensus.cc:3037] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:51:18.568115 17475 raft_consensus.cc:686] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:18.575663 16575 leader_election.cc:302] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:18.594313 17481 raft_consensus.cc:2726] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:51:18.569015 17475 consensus_queue.cc:237] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:18.621513 17025 consensus_queue.cc:1035] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:51:18.643101 16701 leader_election.cc:302] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:18.649056 17349 raft_consensus.cc:2726] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 18:51:18.678969 16338 catalog_manager.cc:5495] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:18.680461 16338 catalog_manager.cc:5495] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:19.025792 17636 consensus_queue.cc:1035] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:51:19.040421 17644 consensus_queue.cc:1035] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:51:19.067122 17644 consensus_queue.cc:1035] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:51:19.079654 17644 consensus_queue.cc:1035] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:51:20.631036 14467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:51:20.638887 17709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:51:20.661623 17710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:51:20.662634 14467 server_base.cc:1034] running on GCE node
W20260810 18:51:20.662914 17712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:51:20.664319 14467 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:51:20.664628 14467 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:51:20.664834 14467 hybrid_clock.cc:648] HybridClock initialized: now 1786387880664818 us; error 0 us; skew 500 ppm
I20260810 18:51:20.665359 14467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:51:20.678128 14467 webserver.cc:460] Webserver started at http://127.14.32.196:46351/ using document root <none> and password file <none>
I20260810 18:51:20.678699 14467 fs_manager.cc:359] Metadata directory not provided
I20260810 18:51:20.678946 14467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:51:20.679370 14467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:51:20.680680 14467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-3-root/instance:
uuid: "831925dd8b644ebab7eb0d672a94eddd"
format_stamp: "Formatted at 2026-08-10 18:51:20 on dist-test-slave-n326"
I20260810 18:51:20.685680 14467 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260810 18:51:20.689975 17720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:51:20.690923 14467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:51:20.691309 14467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-3-root
uuid: "831925dd8b644ebab7eb0d672a94eddd"
format_stamp: "Formatted at 2026-08-10 18:51:20 on dist-test-slave-n326"
I20260810 18:51:20.691607 14467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:51:20.717833 14467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:51:20.719568 14467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:51:20.721524 14467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:51:20.728068 14467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:51:20.728334 14467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:51:20.728641 14467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:51:20.728838 14467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:51:20.789976 14467 rpc_server.cc:307] RPC server started. Bound to: 127.14.32.196:33387
I20260810 18:51:20.790086 17827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.32.196:33387 every 8 connection(s)
I20260810 18:51:20.795593 14467 auto_leader_rebalancer-test.cc:413] add a tserver: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:20.812206 17829 heartbeater.cc:347] Connected to a master server at 127.14.32.254:46351
I20260810 18:51:20.812718 17829 heartbeater.cc:464] Registering TS with master...
I20260810 18:51:20.813501 17829 heartbeater.cc:511] Master 127.14.32.254:46351 requested a full tablet report, sending...
I20260810 18:51:20.815623 16344 ts_manager.cc:194] Registered new tserver with Master: 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:20.817431 16344 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55260
I20260810 18:51:21.524703 16507 consensus_queue.cc:237] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:21.531916 16743 raft_consensus.cc:1261] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:21.533522 17839 consensus_queue.cc:1035] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:21.535192 16624 raft_consensus.cc:1261] T 9e960488563344af83b979c260c2dd06 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:21.537292 17839 consensus_queue.cc:1035] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:21.558177 17362 raft_consensus.cc:2932] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:21.560544 16624 raft_consensus.cc:2932] T 9e960488563344af83b979c260c2dd06 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
W20260810 18:51:21.566079 16447 consensus_peers.cc:487] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea -> Peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387): Couldn't send request to peer 831925dd8b644ebab7eb0d672a94eddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e960488563344af83b979c260c2dd06. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:21.576063 16743 raft_consensus.cc:2932] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:21.584434 16506 consensus_queue.cc:237] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:21.584558 16344 catalog_manager.cc:5495] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea reported cstate change: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260810 18:51:21.602105 16447 consensus_peers.cc:487] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea -> Peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387): Couldn't send request to peer 831925dd8b644ebab7eb0d672a94eddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ccee0891d8df4f81beb2cfbe1b5485f2. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:21.604475 16624 raft_consensus.cc:1261] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:21.605788 17839 consensus_queue.cc:1035] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:21.607635 16743 raft_consensus.cc:1261] T ccee0891d8df4f81beb2cfbe1b5485f2 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:21.609033 17839 consensus_queue.cc:1035] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:21.618814 17452 raft_consensus.cc:2932] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:21.620357 16624 raft_consensus.cc:2932] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:21.632903 16743 raft_consensus.cc:2932] T ccee0891d8df4f81beb2cfbe1b5485f2 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:21.650306 16338 catalog_manager.cc:5495] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:21.668529 16743 consensus_queue.cc:237] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:21.678262 16636 raft_consensus.cc:1261] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:21.681816 17349 consensus_queue.cc:1035] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:21.705797 16506 raft_consensus.cc:1261] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:21.707213 17838 consensus_queue.cc:1035] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:51:21.710180 17349 raft_consensus.cc:2932] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:21.712760 16624 raft_consensus.cc:2932] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:21.733754 16507 consensus_queue.cc:237] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:21.736295 16345 catalog_manager.cc:5495] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260810 18:51:21.740985 16695 consensus_peers.cc:487] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be -> Peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387): Couldn't send request to peer 831925dd8b644ebab7eb0d672a94eddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04ec62e59fcf42ddaf65f9e6b21b49e6. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:21.746726 16506 raft_consensus.cc:2932] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
W20260810 18:51:21.758725 16447 consensus_peers.cc:487] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea -> Peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387): Couldn't send request to peer 831925dd8b644ebab7eb0d672a94eddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f250ae6ba6aa4e55b02199717a2bcde3. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:21.760452 16750 raft_consensus.cc:1261] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:21.762953 17875 consensus_queue.cc:1035] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:51:21.770697 16624 raft_consensus.cc:1261] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:21.772238 17875 consensus_queue.cc:1035] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:51:21.801630 17362 raft_consensus.cc:2932] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:21.804549 16746 raft_consensus.cc:2932] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:21.812731 16622 raft_consensus.cc:2932] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:21.821075 17829 heartbeater.cc:503] Master 127.14.32.254:46351 was elected leader, sending a full tablet report...
I20260810 18:51:21.829519 16344 catalog_manager.cc:5495] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:21.998880 17890 ts_tablet_manager.cc:927] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd: Initiating tablet copy from peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791)
I20260810 18:51:22.020519 17890 tablet_copy_client.cc:316] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Beginning tablet copy session from remote peer at address 127.14.32.193:40791
I20260810 18:51:22.060146 16525 tablet_copy_service.cc:140] P d304ac47fdf846de81337057a4ef7bea: Received BeginTabletCopySession request for tablet ccee0891d8df4f81beb2cfbe1b5485f2 from peer 831925dd8b644ebab7eb0d672a94eddd ({username='slave'} at 127.0.0.1:56468)
I20260810 18:51:22.061058 16525 tablet_copy_service.cc:161] P d304ac47fdf846de81337057a4ef7bea: Beginning new tablet copy session on tablet ccee0891d8df4f81beb2cfbe1b5485f2 from peer 831925dd8b644ebab7eb0d672a94eddd at {username='slave'} at 127.0.0.1:56468: session id = 831925dd8b644ebab7eb0d672a94eddd-ccee0891d8df4f81beb2cfbe1b5485f2
I20260810 18:51:22.076151 16525 tablet_copy_source_session.cc:205] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:51:22.082337 17890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccee0891d8df4f81beb2cfbe1b5485f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:22.111351 17890 tablet_copy_client.cc:791] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 0 data blocks...
I20260810 18:51:22.112246 17890 tablet_copy_client.cc:655] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 1 WAL segments...
I20260810 18:51:22.117962 17890 tablet_copy_client.cc:531] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:51:22.130256 17890 tablet_bootstrap.cc:492] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap starting.
I20260810 18:51:22.132730 17896 ts_tablet_manager.cc:927] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd: Initiating tablet copy from peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791)
I20260810 18:51:22.138269 17896 tablet_copy_client.cc:316] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Beginning tablet copy session from remote peer at address 127.14.32.193:40791
I20260810 18:51:22.139966 16525 tablet_copy_service.cc:140] P d304ac47fdf846de81337057a4ef7bea: Received BeginTabletCopySession request for tablet 9e960488563344af83b979c260c2dd06 from peer 831925dd8b644ebab7eb0d672a94eddd ({username='slave'} at 127.0.0.1:56468)
I20260810 18:51:22.140815 16525 tablet_copy_service.cc:161] P d304ac47fdf846de81337057a4ef7bea: Beginning new tablet copy session on tablet 9e960488563344af83b979c260c2dd06 from peer 831925dd8b644ebab7eb0d672a94eddd at {username='slave'} at 127.0.0.1:56468: session id = 831925dd8b644ebab7eb0d672a94eddd-9e960488563344af83b979c260c2dd06
I20260810 18:51:22.157234 16525 tablet_copy_source_session.cc:205] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:51:22.161928 17896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e960488563344af83b979c260c2dd06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:22.188102 17896 tablet_copy_client.cc:791] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 0 data blocks...
I20260810 18:51:22.190105 17896 tablet_copy_client.cc:655] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 1 WAL segments...
I20260810 18:51:22.190160 17903 ts_tablet_manager.cc:927] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd: Initiating tablet copy from peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791)
I20260810 18:51:22.186856 17890 tablet_bootstrap.cc:492] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:51:22.196620 17890 tablet_bootstrap.cc:492] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap complete.
I20260810 18:51:22.197678 17896 tablet_copy_client.cc:531] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:51:22.198623 17890 ts_tablet_manager.cc:1397] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd: Time spent bootstrapping tablet: real 0.069s user 0.024s sys 0.018s
I20260810 18:51:22.213905 17890 raft_consensus.cc:348] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:22.215351 17890 raft_consensus.cc:729] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Initialized, Role: LEARNER
I20260810 18:51:22.219658 17896 tablet_bootstrap.cc:492] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap starting.
I20260810 18:51:22.216459 17890 consensus_queue.cc:260] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:22.230165 17903 tablet_copy_client.cc:316] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Beginning tablet copy session from remote peer at address 127.14.32.193:40791
I20260810 18:51:22.236836 16525 tablet_copy_service.cc:140] P d304ac47fdf846de81337057a4ef7bea: Received BeginTabletCopySession request for tablet f250ae6ba6aa4e55b02199717a2bcde3 from peer 831925dd8b644ebab7eb0d672a94eddd ({username='slave'} at 127.0.0.1:56468)
I20260810 18:51:22.237273 17890 ts_tablet_manager.cc:1428] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd: Time spent starting tablet: real 0.030s user 0.013s sys 0.011s
I20260810 18:51:22.237463 16525 tablet_copy_service.cc:161] P d304ac47fdf846de81337057a4ef7bea: Beginning new tablet copy session on tablet f250ae6ba6aa4e55b02199717a2bcde3 from peer 831925dd8b644ebab7eb0d672a94eddd at {username='slave'} at 127.0.0.1:56468: session id = 831925dd8b644ebab7eb0d672a94eddd-f250ae6ba6aa4e55b02199717a2bcde3
I20260810 18:51:22.241974 16524 tablet_copy_service.cc:342] P d304ac47fdf846de81337057a4ef7bea: Request end of tablet copy session 831925dd8b644ebab7eb0d672a94eddd-ccee0891d8df4f81beb2cfbe1b5485f2 received from {username='slave'} at 127.0.0.1:56468
I20260810 18:51:22.242447 16524 tablet_copy_service.cc:433] P d304ac47fdf846de81337057a4ef7bea: ending tablet copy session 831925dd8b644ebab7eb0d672a94eddd-ccee0891d8df4f81beb2cfbe1b5485f2 on tablet ccee0891d8df4f81beb2cfbe1b5485f2 with peer 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:22.251675 17905 ts_tablet_manager.cc:927] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd: Initiating tablet copy from peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:22.255419 16525 tablet_copy_source_session.cc:205] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:51:22.261641 17903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f250ae6ba6aa4e55b02199717a2bcde3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:22.278973 17905 tablet_copy_client.cc:316] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Beginning tablet copy session from remote peer at address 127.14.32.195:40171
I20260810 18:51:22.288862 17903 tablet_copy_client.cc:791] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 0 data blocks...
I20260810 18:51:22.289655 17903 tablet_copy_client.cc:655] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 1 WAL segments...
I20260810 18:51:22.297652 17903 tablet_copy_client.cc:531] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:51:22.312340 17903 tablet_bootstrap.cc:492] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap starting.
I20260810 18:51:22.326730 17896 tablet_bootstrap.cc:492] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:51:22.333745 16764 tablet_copy_service.cc:140] P 980f343dd1fd4e049143f8b9d9f2f1be: Received BeginTabletCopySession request for tablet 04ec62e59fcf42ddaf65f9e6b21b49e6 from peer 831925dd8b644ebab7eb0d672a94eddd ({username='slave'} at 127.0.0.1:37716)
I20260810 18:51:22.335603 16764 tablet_copy_service.cc:161] P 980f343dd1fd4e049143f8b9d9f2f1be: Beginning new tablet copy session on tablet 04ec62e59fcf42ddaf65f9e6b21b49e6 from peer 831925dd8b644ebab7eb0d672a94eddd at {username='slave'} at 127.0.0.1:37716: session id = 831925dd8b644ebab7eb0d672a94eddd-04ec62e59fcf42ddaf65f9e6b21b49e6
I20260810 18:51:22.342104 17896 tablet_bootstrap.cc:492] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap complete.
I20260810 18:51:22.343467 17896 ts_tablet_manager.cc:1397] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd: Time spent bootstrapping tablet: real 0.124s user 0.034s sys 0.003s
I20260810 18:51:22.348510 17896 raft_consensus.cc:348] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:22.349678 17896 raft_consensus.cc:729] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Initialized, Role: LEARNER
I20260810 18:51:22.350652 17896 consensus_queue.cc:260] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:22.352507 17903 tablet_bootstrap.cc:492] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:51:22.353394 17903 tablet_bootstrap.cc:492] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap complete.
I20260810 18:51:22.354027 17903 ts_tablet_manager.cc:1397] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd: Time spent bootstrapping tablet: real 0.042s user 0.033s sys 0.003s
I20260810 18:51:22.354970 17896 ts_tablet_manager.cc:1428] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd: Time spent starting tablet: real 0.011s user 0.006s sys 0.003s
I20260810 18:51:22.362731 16525 tablet_copy_service.cc:342] P d304ac47fdf846de81337057a4ef7bea: Request end of tablet copy session 831925dd8b644ebab7eb0d672a94eddd-9e960488563344af83b979c260c2dd06 received from {username='slave'} at 127.0.0.1:56468
I20260810 18:51:22.363195 16525 tablet_copy_service.cc:433] P d304ac47fdf846de81337057a4ef7bea: ending tablet copy session 831925dd8b644ebab7eb0d672a94eddd-9e960488563344af83b979c260c2dd06 on tablet 9e960488563344af83b979c260c2dd06 with peer 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:22.378188 16764 tablet_copy_source_session.cc:205] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:51:22.381120 17903 raft_consensus.cc:348] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:22.382339 17903 raft_consensus.cc:729] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Initialized, Role: LEARNER
I20260810 18:51:22.383170 17905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04ec62e59fcf42ddaf65f9e6b21b49e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:22.383455 17903 consensus_queue.cc:260] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:22.392302 17903 ts_tablet_manager.cc:1428] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd: Time spent starting tablet: real 0.038s user 0.001s sys 0.012s
I20260810 18:51:22.398725 16525 tablet_copy_service.cc:342] P d304ac47fdf846de81337057a4ef7bea: Request end of tablet copy session 831925dd8b644ebab7eb0d672a94eddd-f250ae6ba6aa4e55b02199717a2bcde3 received from {username='slave'} at 127.0.0.1:56468
I20260810 18:51:22.399468 16525 tablet_copy_service.cc:433] P d304ac47fdf846de81337057a4ef7bea: ending tablet copy session 831925dd8b644ebab7eb0d672a94eddd-f250ae6ba6aa4e55b02199717a2bcde3 on tablet f250ae6ba6aa4e55b02199717a2bcde3 with peer 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:22.417116 17905 tablet_copy_client.cc:791] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 0 data blocks...
I20260810 18:51:22.417850 17905 tablet_copy_client.cc:655] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 1 WAL segments...
I20260810 18:51:22.422348 17905 tablet_copy_client.cc:531] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:51:22.434976 17905 tablet_bootstrap.cc:492] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap starting.
I20260810 18:51:22.451908 17786 raft_consensus.cc:1203] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260810 18:51:22.483704 17905 tablet_bootstrap.cc:492] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:51:22.484985 17905 tablet_bootstrap.cc:492] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap complete.
I20260810 18:51:22.485874 17905 ts_tablet_manager.cc:1397] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd: Time spent bootstrapping tablet: real 0.051s user 0.025s sys 0.023s
I20260810 18:51:22.489173 17905 raft_consensus.cc:348] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:22.490054 17905 raft_consensus.cc:729] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Initialized, Role: LEARNER
I20260810 18:51:22.490675 17905 consensus_queue.cc:260] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:22.493306 17905 ts_tablet_manager.cc:1428] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd: Time spent starting tablet: real 0.007s user 0.005s sys 0.004s
I20260810 18:51:22.495254 16764 tablet_copy_service.cc:342] P 980f343dd1fd4e049143f8b9d9f2f1be: Request end of tablet copy session 831925dd8b644ebab7eb0d672a94eddd-04ec62e59fcf42ddaf65f9e6b21b49e6 received from {username='slave'} at 127.0.0.1:37716
I20260810 18:51:22.495642 16764 tablet_copy_service.cc:433] P 980f343dd1fd4e049143f8b9d9f2f1be: ending tablet copy session 831925dd8b644ebab7eb0d672a94eddd-04ec62e59fcf42ddaf65f9e6b21b49e6 on tablet 04ec62e59fcf42ddaf65f9e6b21b49e6 with peer 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:22.619607 17786 raft_consensus.cc:1203] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260810 18:51:22.672839 17786 raft_consensus.cc:1203] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260810 18:51:22.682123 17786 raft_consensus.cc:1203] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260810 18:51:22.926124 17362 raft_consensus.cc:1050] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea: attempting to promote NON_VOTER 831925dd8b644ebab7eb0d672a94eddd to VOTER
I20260810 18:51:22.928308 17362 consensus_queue.cc:237] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:22.934417 17786 raft_consensus.cc:1261] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:22.935643 17839 consensus_queue.cc:1035] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:22.938318 16746 raft_consensus.cc:1261] T ccee0891d8df4f81beb2cfbe1b5485f2 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:22.940878 17875 consensus_queue.cc:1035] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:22.942806 16624 raft_consensus.cc:1261] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:22.947208 17875 consensus_queue.cc:1035] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:22.964926 17839 raft_consensus.cc:2932] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:22.976785 17786 raft_consensus.cc:2932] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:22.989060 16750 raft_consensus.cc:2932] T ccee0891d8df4f81beb2cfbe1b5485f2 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:22.992237 16338 catalog_manager.cc:5495] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.002933 16624 raft_consensus.cc:2932] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.061415 16507 consensus_queue.cc:237] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:23.069811 16624 raft_consensus.cc:1261] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:23.070161 17786 raft_consensus.cc:1261] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:23.071480 17839 consensus_queue.cc:1035] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:51:23.072253 17928 consensus_queue.cc:1035] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:51:23.080861 17928 raft_consensus.cc:2932] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.087477 17786 raft_consensus.cc:2932] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.088469 17926 raft_consensus.cc:1050] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea: attempting to promote NON_VOTER 831925dd8b644ebab7eb0d672a94eddd to VOTER
I20260810 18:51:23.089402 16636 raft_consensus.cc:2932] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.091439 16323 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet ccee0891d8df4f81beb2cfbe1b5485f2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 18:51:23.095052 17926 consensus_queue.cc:237] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:23.102144 16345 catalog_manager.cc:5495] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea reported cstate change: config changed from index 3 to 4, VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) evicted. New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:51:23.113662 16746 raft_consensus.cc:1261] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:23.114346 16726 tablet_service.cc:1514] Processing DeleteTablet for tablet ccee0891d8df4f81beb2cfbe1b5485f2 with delete_type TABLET_DATA_TOMBSTONED (TS 980f343dd1fd4e049143f8b9d9f2f1be not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43720
I20260810 18:51:23.115258 17786 raft_consensus.cc:1261] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:23.115711 17362 consensus_queue.cc:1035] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:23.118104 16636 raft_consensus.cc:1261] T 9e960488563344af83b979c260c2dd06 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:23.117383 17362 consensus_queue.cc:1035] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:23.123986 17926 consensus_queue.cc:1035] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:23.126178 17940 tablet_replica.cc:330] T ccee0891d8df4f81beb2cfbe1b5485f2 P 980f343dd1fd4e049143f8b9d9f2f1be: stopping tablet replica
I20260810 18:51:23.133762 17940 raft_consensus.cc:2229] T ccee0891d8df4f81beb2cfbe1b5485f2 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:23.134107 17839 raft_consensus.cc:1050] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea: attempting to promote NON_VOTER 831925dd8b644ebab7eb0d672a94eddd to VOTER
I20260810 18:51:23.139292 17940 raft_consensus.cc:2258] T ccee0891d8df4f81beb2cfbe1b5485f2 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:23.140410 17839 consensus_queue.cc:237] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:23.152176 16624 raft_consensus.cc:2932] T 9e960488563344af83b979c260c2dd06 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.152395 17924 raft_consensus.cc:2932] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.158412 17940 ts_tablet_manager.cc:1905] T ccee0891d8df4f81beb2cfbe1b5485f2 P 980f343dd1fd4e049143f8b9d9f2f1be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 18:51:23.158480 17786 raft_consensus.cc:2932] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.171201 16345 catalog_manager.cc:5495] T 9e960488563344af83b979c260c2dd06 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.172181 16746 raft_consensus.cc:2932] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.185997 17787 raft_consensus.cc:1261] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:23.186571 16624 raft_consensus.cc:1261] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:23.187932 17928 consensus_queue.cc:1035] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:23.189788 17928 consensus_queue.cc:1035] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:23.191907 16750 raft_consensus.cc:1261] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:23.196583 17928 consensus_queue.cc:1035] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:23.203042 16507 consensus_queue.cc:237] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:23.204486 17940 ts_tablet_manager.cc:1918] T ccee0891d8df4f81beb2cfbe1b5485f2 P 980f343dd1fd4e049143f8b9d9f2f1be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260810 18:51:23.205017 17940 log.cc:1196] T ccee0891d8df4f81beb2cfbe1b5485f2 P 980f343dd1fd4e049143f8b9d9f2f1be: Deleting WAL directory at /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-2-root/wals/ccee0891d8df4f81beb2cfbe1b5485f2
I20260810 18:51:23.208235 16328 catalog_manager.cc:4841] TS 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): tablet ccee0891d8df4f81beb2cfbe1b5485f2 (table test-workload [id=5aa4952cbee8447ca410249e65af5af6]) successfully deleted
I20260810 18:51:23.208899 17787 raft_consensus.cc:1261] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:23.211999 17839 consensus_queue.cc:1035] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:51:23.226719 17881 raft_consensus.cc:1050] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be: attempting to promote NON_VOTER 831925dd8b644ebab7eb0d672a94eddd to VOTER
I20260810 18:51:23.226019 17839 raft_consensus.cc:2932] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.229130 17881 consensus_queue.cc:237] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:23.229256 17785 raft_consensus.cc:2932] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.233417 17924 raft_consensus.cc:2932] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.235050 17786 raft_consensus.cc:2932] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.239012 16323 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 9e960488563344af83b979c260c2dd06 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 18:51:23.245183 16743 raft_consensus.cc:1261] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:23.247663 17839 consensus_queue.cc:1035] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:51:23.257318 16342 catalog_manager.cc:5495] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.260823 16743 raft_consensus.cc:2932] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.268576 16624 raft_consensus.cc:1261] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:23.270867 16605 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e960488563344af83b979c260c2dd06 with delete_type TABLET_DATA_TOMBSTONED (TS 0397d760589d40d3b92ef141cdfc2dbb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53868
I20260810 18:51:23.274293 17786 raft_consensus.cc:1261] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:23.275241 16624 raft_consensus.cc:2932] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.276225 16344 catalog_manager.cc:5495] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.277452 17838 consensus_queue.cc:1035] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:23.279250 17838 consensus_queue.cc:1035] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:23.279500 16507 raft_consensus.cc:1261] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:23.273802 16749 raft_consensus.cc:2932] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.285228 17349 consensus_queue.cc:1035] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:23.317490 17956 tablet_replica.cc:330] T 9e960488563344af83b979c260c2dd06 P 0397d760589d40d3b92ef141cdfc2dbb: stopping tablet replica
I20260810 18:51:23.323889 16501 consensus_queue.cc:237] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:23.336725 16749 raft_consensus.cc:1261] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:23.336977 17785 raft_consensus.cc:1261] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:23.341358 17839 consensus_queue.cc:1035] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:51:23.343847 17924 consensus_queue.cc:1035] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:51:23.344429 17956 raft_consensus.cc:2229] T 9e960488563344af83b979c260c2dd06 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:23.352380 17956 raft_consensus.cc:2258] T 9e960488563344af83b979c260c2dd06 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:23.356549 17349 raft_consensus.cc:2932] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.357371 16501 raft_consensus.cc:2932] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.362442 17928 raft_consensus.cc:2932] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.368283 17956 ts_tablet_manager.cc:1905] T 9e960488563344af83b979c260c2dd06 P 0397d760589d40d3b92ef141cdfc2dbb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 18:51:23.373158 16743 raft_consensus.cc:2932] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.381698 16323 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet f250ae6ba6aa4e55b02199717a2bcde3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 18:51:23.377446 17785 raft_consensus.cc:2932] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.387174 17786 raft_consensus.cc:2932] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.401855 17956 ts_tablet_manager.cc:1918] T 9e960488563344af83b979c260c2dd06 P 0397d760589d40d3b92ef141cdfc2dbb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260810 18:51:23.403345 17956 log.cc:1196] T 9e960488563344af83b979c260c2dd06 P 0397d760589d40d3b92ef141cdfc2dbb: Deleting WAL directory at /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-1-root/wals/9e960488563344af83b979c260c2dd06
I20260810 18:51:23.409968 16324 catalog_manager.cc:4841] TS 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): tablet 9e960488563344af83b979c260c2dd06 (table test-workload [id=5aa4952cbee8447ca410249e65af5af6]) successfully deleted
I20260810 18:51:23.412153 16344 catalog_manager.cc:5495] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea reported cstate change: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:51:23.428087 16636 raft_consensus.cc:2932] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.444391 16342 catalog_manager.cc:5495] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260810 18:51:23.458038 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 72717.1us
I20260810 18:51:23.456676 16605 tablet_service.cc:1514] Processing DeleteTablet for tablet f250ae6ba6aa4e55b02199717a2bcde3 with delete_type TABLET_DATA_TOMBSTONED (TS 0397d760589d40d3b92ef141cdfc2dbb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53868
I20260810 18:51:23.470419 17956 tablet_replica.cc:330] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb: stopping tablet replica
I20260810 18:51:23.471601 17956 raft_consensus.cc:2229] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:23.472398 17956 raft_consensus.cc:2258] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 18:51:23.473681 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 44037.2us
I20260810 18:51:23.479506 16750 consensus_queue.cc:237] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:23.489476 17785 raft_consensus.cc:1261] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:23.490984 17349 consensus_queue.cc:1035] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:51:23.506247 16624 raft_consensus.cc:1261] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:23.505954 17785 raft_consensus.cc:2932] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.502115 17963 raft_consensus.cc:2932] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.514241 17349 consensus_queue.cc:1035] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:51:23.518479 17956 ts_tablet_manager.cc:1905] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 18:51:23.519830 16624 raft_consensus.cc:2932] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.542639 16342 catalog_manager.cc:5495] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:23.552325 16328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 04ec62e59fcf42ddaf65f9e6b21b49e6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 18:51:23.563984 16492 tablet_service.cc:1514] Processing DeleteTablet for tablet 04ec62e59fcf42ddaf65f9e6b21b49e6 with delete_type TABLET_DATA_TOMBSTONED (TS d304ac47fdf846de81337057a4ef7bea not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52330
I20260810 18:51:23.567183 17971 tablet_replica.cc:330] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P d304ac47fdf846de81337057a4ef7bea: stopping tablet replica
I20260810 18:51:23.567989 17971 raft_consensus.cc:2229] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:23.568061 17956 ts_tablet_manager.cc:1918] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260810 18:51:23.568548 17971 raft_consensus.cc:2258] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:23.568763 17956 log.cc:1196] T f250ae6ba6aa4e55b02199717a2bcde3 P 0397d760589d40d3b92ef141cdfc2dbb: Deleting WAL directory at /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-1-root/wals/f250ae6ba6aa4e55b02199717a2bcde3
I20260810 18:51:23.570645 16324 catalog_manager.cc:4841] TS 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): tablet f250ae6ba6aa4e55b02199717a2bcde3 (table test-workload [id=5aa4952cbee8447ca410249e65af5af6]) successfully deleted
I20260810 18:51:23.571943 17971 ts_tablet_manager.cc:1905] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P d304ac47fdf846de81337057a4ef7bea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 18:51:23.590385 17971 ts_tablet_manager.cc:1918] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P d304ac47fdf846de81337057a4ef7bea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260810 18:51:23.590902 17971 log.cc:1196] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P d304ac47fdf846de81337057a4ef7bea: Deleting WAL directory at /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-0-root/wals/04ec62e59fcf42ddaf65f9e6b21b49e6
I20260810 18:51:23.592864 16323 catalog_manager.cc:4841] TS d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): tablet 04ec62e59fcf42ddaf65f9e6b21b49e6 (table test-workload [id=5aa4952cbee8447ca410249e65af5af6]) successfully deleted
I20260810 18:51:24.669832 16506 consensus_queue.cc:237] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:24.677157 16746 raft_consensus.cc:1261] T c758b1e1139941a9a8c4e703cc058061 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260810 18:51:24.677606 16447 consensus_peers.cc:487] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea -> Peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387): Couldn't send request to peer 831925dd8b644ebab7eb0d672a94eddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c758b1e1139941a9a8c4e703cc058061. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:24.678135 16624 raft_consensus.cc:1261] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:24.678960 17977 consensus_queue.cc:1035] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:51:24.680233 17975 consensus_queue.cc:1035] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:51:24.688930 17975 raft_consensus.cc:2932] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:24.690783 16624 raft_consensus.cc:2932] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:24.691956 16746 raft_consensus.cc:2932] T c758b1e1139941a9a8c4e703cc058061 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:24.706022 16750 consensus_queue.cc:237] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:24.714056 16501 raft_consensus.cc:1261] T c3b33c2c321842498975ee1086991616 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:24.714512 16624 raft_consensus.cc:1261] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:24.715883 17349 consensus_queue.cc:1035] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260810 18:51:24.717533 16695 consensus_peers.cc:487] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be -> Peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387): Couldn't send request to peer 831925dd8b644ebab7eb0d672a94eddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3b33c2c321842498975ee1086991616. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:24.718757 17881 consensus_queue.cc:1035] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:51:24.725044 16345 catalog_manager.cc:5495] T c758b1e1139941a9a8c4e703cc058061 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:24.731235 17349 raft_consensus.cc:2932] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:24.732810 16622 raft_consensus.cc:2932] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:24.777029 16506 raft_consensus.cc:2932] T c3b33c2c321842498975ee1086991616 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:24.784730 16624 consensus_queue.cc:237] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:24.807873 16345 catalog_manager.cc:5495] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:24.810827 16750 raft_consensus.cc:1261] T ab9362a1df854c4fad0204ee2f9f78ad P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:24.811146 16506 raft_consensus.cc:1261] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:24.812706 17967 consensus_queue.cc:1035] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:51:24.814352 17970 consensus_queue.cc:1035] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:51:24.830309 17970 raft_consensus.cc:2932] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
W20260810 18:51:24.833003 16573 consensus_peers.cc:487] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb -> Peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387): Couldn't send request to peer 831925dd8b644ebab7eb0d672a94eddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab9362a1df854c4fad0204ee2f9f78ad. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:24.835651 16506 raft_consensus.cc:2932] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:24.838057 16750 raft_consensus.cc:2932] T ab9362a1df854c4fad0204ee2f9f78ad P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:24.850945 16501 consensus_queue.cc:237] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:24.867568 16750 raft_consensus.cc:1261] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:24.870702 17977 consensus_queue.cc:1035] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:51:24.863556 16345 catalog_manager.cc:5495] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260810 18:51:24.874856 16447 consensus_peers.cc:487] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea -> Peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387): Couldn't send request to peer 831925dd8b644ebab7eb0d672a94eddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 694fa0ce1cd246a985705792780adee0. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:24.881592 16624 raft_consensus.cc:1261] T 694fa0ce1cd246a985705792780adee0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:24.883301 17975 consensus_queue.cc:1035] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:24.885238 17924 raft_consensus.cc:2932] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:24.887732 16746 raft_consensus.cc:2932] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:24.906327 16624 raft_consensus.cc:2932] T 694fa0ce1cd246a985705792780adee0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:24.907092 16342 catalog_manager.cc:5495] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:25.180143 18014 ts_tablet_manager.cc:927] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd: Initiating tablet copy from peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791)
I20260810 18:51:25.182519 18014 tablet_copy_client.cc:316] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Beginning tablet copy session from remote peer at address 127.14.32.193:40791
I20260810 18:51:25.184448 16525 tablet_copy_service.cc:140] P d304ac47fdf846de81337057a4ef7bea: Received BeginTabletCopySession request for tablet c758b1e1139941a9a8c4e703cc058061 from peer 831925dd8b644ebab7eb0d672a94eddd ({username='slave'} at 127.0.0.1:56468)
I20260810 18:51:25.184980 16525 tablet_copy_service.cc:161] P d304ac47fdf846de81337057a4ef7bea: Beginning new tablet copy session on tablet c758b1e1139941a9a8c4e703cc058061 from peer 831925dd8b644ebab7eb0d672a94eddd at {username='slave'} at 127.0.0.1:56468: session id = 831925dd8b644ebab7eb0d672a94eddd-c758b1e1139941a9a8c4e703cc058061
I20260810 18:51:25.192639 16525 tablet_copy_source_session.cc:205] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:51:25.195549 18014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c758b1e1139941a9a8c4e703cc058061. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:25.209583 18014 tablet_copy_client.cc:791] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 0 data blocks...
I20260810 18:51:25.210274 18014 tablet_copy_client.cc:655] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 1 WAL segments...
I20260810 18:51:25.214448 18014 tablet_copy_client.cc:531] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:51:25.222724 18014 tablet_bootstrap.cc:492] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap starting.
I20260810 18:51:25.228006 18016 ts_tablet_manager.cc:927] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd: Initiating tablet copy from peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:25.232018 18016 tablet_copy_client.cc:316] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Beginning tablet copy session from remote peer at address 127.14.32.195:40171
I20260810 18:51:25.238255 16764 tablet_copy_service.cc:140] P 980f343dd1fd4e049143f8b9d9f2f1be: Received BeginTabletCopySession request for tablet c3b33c2c321842498975ee1086991616 from peer 831925dd8b644ebab7eb0d672a94eddd ({username='slave'} at 127.0.0.1:37716)
I20260810 18:51:25.238992 16764 tablet_copy_service.cc:161] P 980f343dd1fd4e049143f8b9d9f2f1be: Beginning new tablet copy session on tablet c3b33c2c321842498975ee1086991616 from peer 831925dd8b644ebab7eb0d672a94eddd at {username='slave'} at 127.0.0.1:37716: session id = 831925dd8b644ebab7eb0d672a94eddd-c3b33c2c321842498975ee1086991616
I20260810 18:51:25.248741 16764 tablet_copy_source_session.cc:205] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:51:25.250573 18014 tablet_bootstrap.cc:492] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:51:25.251631 18014 tablet_bootstrap.cc:492] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap complete.
I20260810 18:51:25.251725 18016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b33c2c321842498975ee1086991616. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:25.252367 18014 ts_tablet_manager.cc:1397] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd: Time spent bootstrapping tablet: real 0.030s user 0.025s sys 0.005s
I20260810 18:51:25.256155 18014 raft_consensus.cc:348] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:25.257081 18014 raft_consensus.cc:729] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Initialized, Role: LEARNER
I20260810 18:51:25.257871 18014 consensus_queue.cc:260] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:25.266975 18016 tablet_copy_client.cc:791] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 0 data blocks...
I20260810 18:51:25.267596 18016 tablet_copy_client.cc:655] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 1 WAL segments...
I20260810 18:51:25.269901 18014 ts_tablet_manager.cc:1428] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd: Time spent starting tablet: real 0.017s user 0.006s sys 0.012s
I20260810 18:51:25.271688 16525 tablet_copy_service.cc:342] P d304ac47fdf846de81337057a4ef7bea: Request end of tablet copy session 831925dd8b644ebab7eb0d672a94eddd-c758b1e1139941a9a8c4e703cc058061 received from {username='slave'} at 127.0.0.1:56468
I20260810 18:51:25.272145 16525 tablet_copy_service.cc:433] P d304ac47fdf846de81337057a4ef7bea: ending tablet copy session 831925dd8b644ebab7eb0d672a94eddd-c758b1e1139941a9a8c4e703cc058061 on tablet c758b1e1139941a9a8c4e703cc058061 with peer 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:25.280385 18016 tablet_copy_client.cc:531] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:51:25.288003 18016 tablet_bootstrap.cc:492] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap starting.
I20260810 18:51:25.306326 18016 tablet_bootstrap.cc:492] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:51:25.307420 18016 tablet_bootstrap.cc:492] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap complete.
I20260810 18:51:25.308077 18016 ts_tablet_manager.cc:1397] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.006s
I20260810 18:51:25.310807 18016 raft_consensus.cc:348] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:25.311563 18016 raft_consensus.cc:729] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Initialized, Role: LEARNER
I20260810 18:51:25.312278 18016 consensus_queue.cc:260] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:25.314687 18014 ts_tablet_manager.cc:927] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd: Initiating tablet copy from peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:25.315053 18016 ts_tablet_manager.cc:1428] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260810 18:51:25.316910 16764 tablet_copy_service.cc:342] P 980f343dd1fd4e049143f8b9d9f2f1be: Request end of tablet copy session 831925dd8b644ebab7eb0d672a94eddd-c3b33c2c321842498975ee1086991616 received from {username='slave'} at 127.0.0.1:37716
I20260810 18:51:25.317304 16764 tablet_copy_service.cc:433] P 980f343dd1fd4e049143f8b9d9f2f1be: ending tablet copy session 831925dd8b644ebab7eb0d672a94eddd-c3b33c2c321842498975ee1086991616 on tablet c3b33c2c321842498975ee1086991616 with peer 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:25.333318 18014 tablet_copy_client.cc:316] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Beginning tablet copy session from remote peer at address 127.14.32.194:45827
I20260810 18:51:25.353176 16648 tablet_copy_service.cc:140] P 0397d760589d40d3b92ef141cdfc2dbb: Received BeginTabletCopySession request for tablet ab9362a1df854c4fad0204ee2f9f78ad from peer 831925dd8b644ebab7eb0d672a94eddd ({username='slave'} at 127.0.0.1:60066)
I20260810 18:51:25.353782 16648 tablet_copy_service.cc:161] P 0397d760589d40d3b92ef141cdfc2dbb: Beginning new tablet copy session on tablet ab9362a1df854c4fad0204ee2f9f78ad from peer 831925dd8b644ebab7eb0d672a94eddd at {username='slave'} at 127.0.0.1:60066: session id = 831925dd8b644ebab7eb0d672a94eddd-ab9362a1df854c4fad0204ee2f9f78ad
I20260810 18:51:25.361418 16648 tablet_copy_source_session.cc:205] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:51:25.364248 18014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab9362a1df854c4fad0204ee2f9f78ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:25.379345 18014 tablet_copy_client.cc:791] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 0 data blocks...
I20260810 18:51:25.379942 18014 tablet_copy_client.cc:655] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 1 WAL segments...
I20260810 18:51:25.384477 18014 tablet_copy_client.cc:531] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:51:25.393990 18014 tablet_bootstrap.cc:492] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap starting.
I20260810 18:51:25.435943 18014 tablet_bootstrap.cc:492] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:51:25.436723 18014 tablet_bootstrap.cc:492] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap complete.
I20260810 18:51:25.437387 18014 ts_tablet_manager.cc:1397] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd: Time spent bootstrapping tablet: real 0.044s user 0.024s sys 0.008s
I20260810 18:51:25.439782 18014 raft_consensus.cc:348] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:25.440683 18014 raft_consensus.cc:729] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Initialized, Role: LEARNER
I20260810 18:51:25.441270 18014 consensus_queue.cc:260] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:25.443346 18014 ts_tablet_manager.cc:1428] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260810 18:51:25.445161 16648 tablet_copy_service.cc:342] P 0397d760589d40d3b92ef141cdfc2dbb: Request end of tablet copy session 831925dd8b644ebab7eb0d672a94eddd-ab9362a1df854c4fad0204ee2f9f78ad received from {username='slave'} at 127.0.0.1:60066
I20260810 18:51:25.445531 16648 tablet_copy_service.cc:433] P 0397d760589d40d3b92ef141cdfc2dbb: ending tablet copy session 831925dd8b644ebab7eb0d672a94eddd-ab9362a1df854c4fad0204ee2f9f78ad on tablet ab9362a1df854c4fad0204ee2f9f78ad with peer 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:25.486063 18014 ts_tablet_manager.cc:927] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd: Initiating tablet copy from peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791)
I20260810 18:51:25.488349 18014 tablet_copy_client.cc:316] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Beginning tablet copy session from remote peer at address 127.14.32.193:40791
I20260810 18:51:25.490429 16525 tablet_copy_service.cc:140] P d304ac47fdf846de81337057a4ef7bea: Received BeginTabletCopySession request for tablet 694fa0ce1cd246a985705792780adee0 from peer 831925dd8b644ebab7eb0d672a94eddd ({username='slave'} at 127.0.0.1:56468)
I20260810 18:51:25.490940 16525 tablet_copy_service.cc:161] P d304ac47fdf846de81337057a4ef7bea: Beginning new tablet copy session on tablet 694fa0ce1cd246a985705792780adee0 from peer 831925dd8b644ebab7eb0d672a94eddd at {username='slave'} at 127.0.0.1:56468: session id = 831925dd8b644ebab7eb0d672a94eddd-694fa0ce1cd246a985705792780adee0
I20260810 18:51:25.499892 16525 tablet_copy_source_session.cc:205] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:51:25.502985 18014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 694fa0ce1cd246a985705792780adee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:25.517258 18014 tablet_copy_client.cc:791] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 0 data blocks...
I20260810 18:51:25.517952 18014 tablet_copy_client.cc:655] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 1 WAL segments...
I20260810 18:51:25.521126 18014 tablet_copy_client.cc:531] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:51:25.529552 18014 tablet_bootstrap.cc:492] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap starting.
I20260810 18:51:25.549986 18014 tablet_bootstrap.cc:492] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:51:25.550830 18014 tablet_bootstrap.cc:492] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap complete.
I20260810 18:51:25.551512 18014 ts_tablet_manager.cc:1397] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd: Time spent bootstrapping tablet: real 0.022s user 0.017s sys 0.003s
I20260810 18:51:25.553668 18014 raft_consensus.cc:348] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:25.554311 18014 raft_consensus.cc:729] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Initialized, Role: LEARNER
I20260810 18:51:25.554870 18014 consensus_queue.cc:260] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:25.556695 18014 ts_tablet_manager.cc:1428] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260810 18:51:25.558997 16525 tablet_copy_service.cc:342] P d304ac47fdf846de81337057a4ef7bea: Request end of tablet copy session 831925dd8b644ebab7eb0d672a94eddd-694fa0ce1cd246a985705792780adee0 received from {username='slave'} at 127.0.0.1:56468
I20260810 18:51:25.559425 16525 tablet_copy_service.cc:433] P d304ac47fdf846de81337057a4ef7bea: ending tablet copy session 831925dd8b644ebab7eb0d672a94eddd-694fa0ce1cd246a985705792780adee0 on tablet 694fa0ce1cd246a985705792780adee0 with peer 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:25.801673 17785 raft_consensus.cc:1203] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260810 18:51:25.819584 17785 raft_consensus.cc:1203] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260810 18:51:25.931288 17785 raft_consensus.cc:1203] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260810 18:51:26.047583 17785 raft_consensus.cc:1203] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260810 18:51:26.192399 18042 raft_consensus.cc:1050] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea: attempting to promote NON_VOTER 831925dd8b644ebab7eb0d672a94eddd to VOTER
I20260810 18:51:26.194561 18042 consensus_queue.cc:237] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:26.206322 16622 raft_consensus.cc:1261] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:26.209236 17785 raft_consensus.cc:1261] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:26.212252 18042 consensus_queue.cc:1035] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:26.214362 18034 consensus_queue.cc:1035] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:26.216109 16746 raft_consensus.cc:1261] T c758b1e1139941a9a8c4e703cc058061 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:26.229787 17924 consensus_queue.cc:1035] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:26.247938 17924 raft_consensus.cc:2932] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.250488 17785 raft_consensus.cc:2932] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.251595 16746 raft_consensus.cc:2932] T c758b1e1139941a9a8c4e703cc058061 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.263727 16342 catalog_manager.cc:5495] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.275806 16624 raft_consensus.cc:2932] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.318244 16505 consensus_queue.cc:237] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:26.325031 16622 raft_consensus.cc:1261] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:26.326380 18042 consensus_queue.cc:1035] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:51:26.326707 17785 raft_consensus.cc:1261] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:26.328121 18034 consensus_queue.cc:1035] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:51:26.333431 17964 raft_consensus.cc:1050] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be: attempting to promote NON_VOTER 831925dd8b644ebab7eb0d672a94eddd to VOTER
I20260810 18:51:26.334486 18042 raft_consensus.cc:2932] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.335747 17964 consensus_queue.cc:237] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:26.336380 16622 raft_consensus.cc:2932] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.341117 17785 raft_consensus.cc:2932] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.352241 18047 raft_consensus.cc:1050] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb: attempting to promote NON_VOTER 831925dd8b644ebab7eb0d672a94eddd to VOTER
I20260810 18:51:26.355738 17786 raft_consensus.cc:1261] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:26.354867 16344 catalog_manager.cc:5495] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: config changed from index 3 to 4, VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) evicted. New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.356797 16501 raft_consensus.cc:1261] T c3b33c2c321842498975ee1086991616 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:26.357638 16622 raft_consensus.cc:1261] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:26.357607 18043 consensus_queue.cc:1035] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:26.357517 18047 consensus_queue.cc:237] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:26.360409 18043 consensus_queue.cc:1035] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:26.363570 18043 consensus_queue.cc:1035] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:26.372361 16726 tablet_service.cc:1514] Processing DeleteTablet for tablet c758b1e1139941a9a8c4e703cc058061 with delete_type TABLET_DATA_TOMBSTONED (TS 980f343dd1fd4e049143f8b9d9f2f1be not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43720
I20260810 18:51:26.383752 16323 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet c758b1e1139941a9a8c4e703cc058061 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 18:51:26.391726 18071 tablet_replica.cc:330] T c758b1e1139941a9a8c4e703cc058061 P 980f343dd1fd4e049143f8b9d9f2f1be: stopping tablet replica
I20260810 18:51:26.393429 18071 raft_consensus.cc:2229] T c758b1e1139941a9a8c4e703cc058061 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:26.394608 18071 raft_consensus.cc:2258] T c758b1e1139941a9a8c4e703cc058061 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:26.404111 17785 raft_consensus.cc:1261] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:26.403788 18043 raft_consensus.cc:2932] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.408378 17970 consensus_queue.cc:1035] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:26.408286 16501 raft_consensus.cc:1261] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:26.419165 18008 consensus_queue.cc:1035] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:26.419927 16750 raft_consensus.cc:1261] T ab9362a1df854c4fad0204ee2f9f78ad P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:26.418289 16622 raft_consensus.cc:2932] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.425426 17967 consensus_queue.cc:1035] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:26.435781 18071 ts_tablet_manager.cc:1905] T c758b1e1139941a9a8c4e703cc058061 P 980f343dd1fd4e049143f8b9d9f2f1be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260810 18:51:26.439853 18061 log.cc:924] Time spent T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.063s user 0.001s sys 0.000s
I20260810 18:51:26.444090 16344 catalog_manager.cc:5495] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.457332 18047 raft_consensus.cc:2932] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.465679 18071 ts_tablet_manager.cc:1918] T c758b1e1139941a9a8c4e703cc058061 P 980f343dd1fd4e049143f8b9d9f2f1be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260810 18:51:26.469035 18071 log.cc:1196] T c758b1e1139941a9a8c4e703cc058061 P 980f343dd1fd4e049143f8b9d9f2f1be: Deleting WAL directory at /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-2-root/wals/c758b1e1139941a9a8c4e703cc058061
I20260810 18:51:26.480501 16328 catalog_manager.cc:4841] TS 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): tablet c758b1e1139941a9a8c4e703cc058061 (table test-workload [id=5aa4952cbee8447ca410249e65af5af6]) successfully deleted
I20260810 18:51:26.480695 16501 raft_consensus.cc:2932] T c3b33c2c321842498975ee1086991616 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.488198 16749 raft_consensus.cc:2932] T ab9362a1df854c4fad0204ee2f9f78ad P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.488279 17785 raft_consensus.cc:2932] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.509482 17786 raft_consensus.cc:2932] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.557204 16344 catalog_manager.cc:5495] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:51:26.567093 16501 raft_consensus.cc:2932] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.600304 18042 raft_consensus.cc:1050] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea: attempting to promote NON_VOTER 831925dd8b644ebab7eb0d672a94eddd to VOTER
I20260810 18:51:26.600260 16624 consensus_queue.cc:237] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:26.602996 16749 consensus_queue.cc:237] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:26.606185 17785 raft_consensus.cc:1261] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:26.607930 18077 consensus_queue.cc:1035] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:51:26.609105 18042 consensus_queue.cc:237] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:26.611991 16505 raft_consensus.cc:1261] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:26.615828 18077 consensus_queue.cc:1035] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:51:26.655323 16749 raft_consensus.cc:1261] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:26.660341 17786 raft_consensus.cc:1261] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:26.653007 17785 raft_consensus.cc:2932] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.661885 17924 consensus_queue.cc:1035] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:26.665833 16631 raft_consensus.cc:1261] T 694fa0ce1cd246a985705792780adee0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:26.667505 17977 consensus_queue.cc:1035] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:26.669287 17977 consensus_queue.cc:1035] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:26.668982 16501 raft_consensus.cc:2932] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.645228 18008 raft_consensus.cc:2932] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.693714 16631 raft_consensus.cc:1261] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:26.693677 16345 catalog_manager.cc:5495] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: config changed from index 3 to 4, VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) evicted. New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.697774 18081 consensus_queue.cc:1035] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:51:26.710836 17964 raft_consensus.cc:2932] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.717473 16622 raft_consensus.cc:2932] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.742067 18042 raft_consensus.cc:2932] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.743955 16344 catalog_manager.cc:5495] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:26.744601 17786 raft_consensus.cc:2932] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.754935 17785 raft_consensus.cc:1261] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:26.755244 16328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet c3b33c2c321842498975ee1086991616 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 18:51:26.759568 16726 tablet_service.cc:1514] Processing DeleteTablet for tablet ab9362a1df854c4fad0204ee2f9f78ad with delete_type TABLET_DATA_TOMBSTONED (TS 980f343dd1fd4e049143f8b9d9f2f1be not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43720
I20260810 18:51:26.757340 16624 raft_consensus.cc:2932] T 694fa0ce1cd246a985705792780adee0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.760490 18071 tablet_replica.cc:330] T ab9362a1df854c4fad0204ee2f9f78ad P 980f343dd1fd4e049143f8b9d9f2f1be: stopping tablet replica
I20260810 18:51:26.761490 18071 raft_consensus.cc:2229] T ab9362a1df854c4fad0204ee2f9f78ad P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:26.762080 18071 raft_consensus.cc:2258] T ab9362a1df854c4fad0204ee2f9f78ad P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:26.760485 18083 consensus_queue.cc:1035] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260810 18:51:26.767500 16492 tablet_service.cc:1514] Processing DeleteTablet for tablet c3b33c2c321842498975ee1086991616 with delete_type TABLET_DATA_TOMBSTONED (TS d304ac47fdf846de81337057a4ef7bea not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52330
I20260810 18:51:26.767956 18071 ts_tablet_manager.cc:1905] T ab9362a1df854c4fad0204ee2f9f78ad P 980f343dd1fd4e049143f8b9d9f2f1be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 18:51:26.768062 16338 catalog_manager.cc:5495] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.784535 16324 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet ab9362a1df854c4fad0204ee2f9f78ad with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 18:51:26.791893 18097 tablet_replica.cc:330] T c3b33c2c321842498975ee1086991616 P d304ac47fdf846de81337057a4ef7bea: stopping tablet replica
I20260810 18:51:26.791450 16749 raft_consensus.cc:2932] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.793321 18097 raft_consensus.cc:2229] T c3b33c2c321842498975ee1086991616 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:26.793215 17786 raft_consensus.cc:2932] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.794224 18097 raft_consensus.cc:2258] T c3b33c2c321842498975ee1086991616 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:26.799285 18097 ts_tablet_manager.cc:1905] T c3b33c2c321842498975ee1086991616 P d304ac47fdf846de81337057a4ef7bea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 18:51:26.814765 18071 ts_tablet_manager.cc:1918] T ab9362a1df854c4fad0204ee2f9f78ad P 980f343dd1fd4e049143f8b9d9f2f1be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260810 18:51:26.815466 18071 log.cc:1196] T ab9362a1df854c4fad0204ee2f9f78ad P 980f343dd1fd4e049143f8b9d9f2f1be: Deleting WAL directory at /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-2-root/wals/ab9362a1df854c4fad0204ee2f9f78ad
I20260810 18:51:26.817276 16328 catalog_manager.cc:4841] TS 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): tablet ab9362a1df854c4fad0204ee2f9f78ad (table test-workload [id=5aa4952cbee8447ca410249e65af5af6]) successfully deleted
I20260810 18:51:26.830323 18097 ts_tablet_manager.cc:1918] T c3b33c2c321842498975ee1086991616 P d304ac47fdf846de81337057a4ef7bea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260810 18:51:26.830998 18097 log.cc:1196] T c3b33c2c321842498975ee1086991616 P d304ac47fdf846de81337057a4ef7bea: Deleting WAL directory at /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-0-root/wals/c3b33c2c321842498975ee1086991616
I20260810 18:51:26.838075 16323 catalog_manager.cc:4841] TS d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): tablet c3b33c2c321842498975ee1086991616 (table test-workload [id=5aa4952cbee8447ca410249e65af5af6]) successfully deleted
I20260810 18:51:26.845781 16506 consensus_queue.cc:237] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:26.855808 16749 raft_consensus.cc:1261] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:26.860618 18042 consensus_queue.cc:1035] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:51:26.860419 17786 raft_consensus.cc:1261] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:26.866370 18042 consensus_queue.cc:1035] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:51:26.868100 17924 raft_consensus.cc:2932] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.873539 16749 raft_consensus.cc:2932] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.874390 17786 raft_consensus.cc:2932] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.883280 16323 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 694fa0ce1cd246a985705792780adee0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 18:51:26.887650 16342 catalog_manager.cc:5495] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:26.899299 16605 tablet_service.cc:1514] Processing DeleteTablet for tablet 694fa0ce1cd246a985705792780adee0 with delete_type TABLET_DATA_TOMBSTONED (TS 0397d760589d40d3b92ef141cdfc2dbb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53868
I20260810 18:51:26.901466 18100 tablet_replica.cc:330] T 694fa0ce1cd246a985705792780adee0 P 0397d760589d40d3b92ef141cdfc2dbb: stopping tablet replica
I20260810 18:51:26.903296 18100 raft_consensus.cc:2229] T 694fa0ce1cd246a985705792780adee0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:26.904126 18100 raft_consensus.cc:2258] T 694fa0ce1cd246a985705792780adee0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:26.910148 18100 ts_tablet_manager.cc:1905] T 694fa0ce1cd246a985705792780adee0 P 0397d760589d40d3b92ef141cdfc2dbb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 18:51:26.926477 18100 ts_tablet_manager.cc:1918] T 694fa0ce1cd246a985705792780adee0 P 0397d760589d40d3b92ef141cdfc2dbb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260810 18:51:26.926839 18100 log.cc:1196] T 694fa0ce1cd246a985705792780adee0 P 0397d760589d40d3b92ef141cdfc2dbb: Deleting WAL directory at /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-1-root/wals/694fa0ce1cd246a985705792780adee0
I20260810 18:51:26.928411 16324 catalog_manager.cc:4841] TS 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): tablet 694fa0ce1cd246a985705792780adee0 (table test-workload [id=5aa4952cbee8447ca410249e65af5af6]) successfully deleted
I20260810 18:51:27.964422 16624 consensus_queue.cc:237] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:27.970052 16746 raft_consensus.cc:1261] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260810 18:51:27.971215 16573 consensus_peers.cc:487] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb -> Peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387): Couldn't send request to peer 831925dd8b644ebab7eb0d672a94eddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: abdb4e808ce3454ab9a52051b369e5d0. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:27.972157 18105 consensus_queue.cc:1035] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:51:27.972604 16501 raft_consensus.cc:1261] T abdb4e808ce3454ab9a52051b369e5d0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:27.973932 18101 consensus_queue.cc:1035] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:27.980058 18105 raft_consensus.cc:2932] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:27.981848 16746 raft_consensus.cc:2932] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:27.984141 16501 raft_consensus.cc:2932] T abdb4e808ce3454ab9a52051b369e5d0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:27.995749 16501 consensus_queue.cc:237] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:27.998529 16345 catalog_manager.cc:5495] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260810 18:51:28.004216 16447 consensus_peers.cc:487] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea -> Peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387): Couldn't send request to peer 831925dd8b644ebab7eb0d672a94eddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 681439c754d942b7847d73744abbe269. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:28.004277 16631 raft_consensus.cc:1261] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:28.007149 16746 raft_consensus.cc:1261] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:28.008581 18102 consensus_queue.cc:1035] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:28.011896 18102 consensus_queue.cc:1035] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:28.027264 17977 raft_consensus.cc:2932] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:28.028821 16749 raft_consensus.cc:2932] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:28.029219 16631 raft_consensus.cc:2932] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:28.066169 16342 catalog_manager.cc:5495] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:28.074854 16631 consensus_queue.cc:237] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:28.090307 16746 raft_consensus.cc:1261] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260810 18:51:28.091044 16573 consensus_peers.cc:487] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb -> Peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387): Couldn't send request to peer 831925dd8b644ebab7eb0d672a94eddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 192ad39135ed48d5adab628ba706791a. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:28.092073 16501 raft_consensus.cc:1261] T 192ad39135ed48d5adab628ba706791a P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:28.091776 18105 consensus_queue.cc:1035] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:51:28.093762 18105 consensus_queue.cc:1035] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:28.100394 18101 raft_consensus.cc:2932] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:28.103091 16501 raft_consensus.cc:2932] T 192ad39135ed48d5adab628ba706791a P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:28.104400 16746 raft_consensus.cc:2932] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:28.118335 16749 consensus_queue.cc:237] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:28.121583 16345 catalog_manager.cc:5495] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:28.136951 16631 raft_consensus.cc:1261] T 42bb15fb54124b8db9b482e3f17fc87c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:28.137193 16505 raft_consensus.cc:1261] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260810 18:51:28.139123 16695 consensus_peers.cc:487] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be -> Peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387): Couldn't send request to peer 831925dd8b644ebab7eb0d672a94eddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42bb15fb54124b8db9b482e3f17fc87c. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:28.139744 18086 consensus_queue.cc:1035] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260810 18:51:28.142127 17964 consensus_queue.cc:1035] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:28.153512 18086 raft_consensus.cc:2932] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:28.155138 16505 raft_consensus.cc:2932] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:28.166216 16624 raft_consensus.cc:2932] T 42bb15fb54124b8db9b482e3f17fc87c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:28.175848 16343 catalog_manager.cc:5495] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:28.364352 18131 ts_tablet_manager.cc:927] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd: Initiating tablet copy from peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:28.366109 18131 tablet_copy_client.cc:316] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Beginning tablet copy session from remote peer at address 127.14.32.194:45827
I20260810 18:51:28.367673 16648 tablet_copy_service.cc:140] P 0397d760589d40d3b92ef141cdfc2dbb: Received BeginTabletCopySession request for tablet abdb4e808ce3454ab9a52051b369e5d0 from peer 831925dd8b644ebab7eb0d672a94eddd ({username='slave'} at 127.0.0.1:60066)
I20260810 18:51:28.368172 16648 tablet_copy_service.cc:161] P 0397d760589d40d3b92ef141cdfc2dbb: Beginning new tablet copy session on tablet abdb4e808ce3454ab9a52051b369e5d0 from peer 831925dd8b644ebab7eb0d672a94eddd at {username='slave'} at 127.0.0.1:60066: session id = 831925dd8b644ebab7eb0d672a94eddd-abdb4e808ce3454ab9a52051b369e5d0
I20260810 18:51:28.375830 16648 tablet_copy_source_session.cc:205] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:51:28.378644 18131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdb4e808ce3454ab9a52051b369e5d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:28.393127 18131 tablet_copy_client.cc:791] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 0 data blocks...
I20260810 18:51:28.393970 18131 tablet_copy_client.cc:655] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 1 WAL segments...
I20260810 18:51:28.397390 18131 tablet_copy_client.cc:531] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:51:28.405532 18131 tablet_bootstrap.cc:492] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap starting.
I20260810 18:51:28.429360 18131 tablet_bootstrap.cc:492] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:51:28.430277 18131 tablet_bootstrap.cc:492] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap complete.
I20260810 18:51:28.430930 18131 ts_tablet_manager.cc:1397] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.017s
I20260810 18:51:28.433317 18131 raft_consensus.cc:348] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:28.434274 18131 raft_consensus.cc:729] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Initialized, Role: LEARNER
I20260810 18:51:28.434847 18131 consensus_queue.cc:260] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:28.438095 18131 ts_tablet_manager.cc:1428] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260810 18:51:28.439685 16648 tablet_copy_service.cc:342] P 0397d760589d40d3b92ef141cdfc2dbb: Request end of tablet copy session 831925dd8b644ebab7eb0d672a94eddd-abdb4e808ce3454ab9a52051b369e5d0 received from {username='slave'} at 127.0.0.1:60066
I20260810 18:51:28.440079 16648 tablet_copy_service.cc:433] P 0397d760589d40d3b92ef141cdfc2dbb: ending tablet copy session 831925dd8b644ebab7eb0d672a94eddd-abdb4e808ce3454ab9a52051b369e5d0 on tablet abdb4e808ce3454ab9a52051b369e5d0 with peer 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:28.470470 18131 ts_tablet_manager.cc:927] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd: Initiating tablet copy from peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:28.474064 18131 tablet_copy_client.cc:316] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Beginning tablet copy session from remote peer at address 127.14.32.194:45827
I20260810 18:51:28.475437 16648 tablet_copy_service.cc:140] P 0397d760589d40d3b92ef141cdfc2dbb: Received BeginTabletCopySession request for tablet 192ad39135ed48d5adab628ba706791a from peer 831925dd8b644ebab7eb0d672a94eddd ({username='slave'} at 127.0.0.1:60066)
I20260810 18:51:28.475874 16648 tablet_copy_service.cc:161] P 0397d760589d40d3b92ef141cdfc2dbb: Beginning new tablet copy session on tablet 192ad39135ed48d5adab628ba706791a from peer 831925dd8b644ebab7eb0d672a94eddd at {username='slave'} at 127.0.0.1:60066: session id = 831925dd8b644ebab7eb0d672a94eddd-192ad39135ed48d5adab628ba706791a
I20260810 18:51:28.488757 16648 tablet_copy_source_session.cc:205] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:51:28.502501 18131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192ad39135ed48d5adab628ba706791a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:28.504043 18140 ts_tablet_manager.cc:927] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd: Initiating tablet copy from peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791)
I20260810 18:51:28.507798 18140 tablet_copy_client.cc:316] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Beginning tablet copy session from remote peer at address 127.14.32.193:40791
I20260810 18:51:28.509364 16525 tablet_copy_service.cc:140] P d304ac47fdf846de81337057a4ef7bea: Received BeginTabletCopySession request for tablet 681439c754d942b7847d73744abbe269 from peer 831925dd8b644ebab7eb0d672a94eddd ({username='slave'} at 127.0.0.1:56468)
I20260810 18:51:28.509925 16525 tablet_copy_service.cc:161] P d304ac47fdf846de81337057a4ef7bea: Beginning new tablet copy session on tablet 681439c754d942b7847d73744abbe269 from peer 831925dd8b644ebab7eb0d672a94eddd at {username='slave'} at 127.0.0.1:56468: session id = 831925dd8b644ebab7eb0d672a94eddd-681439c754d942b7847d73744abbe269
I20260810 18:51:28.520282 18131 tablet_copy_client.cc:791] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 0 data blocks...
I20260810 18:51:28.521143 18131 tablet_copy_client.cc:655] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 1 WAL segments...
I20260810 18:51:28.521822 16525 tablet_copy_source_session.cc:205] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:51:28.524899 18140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 681439c754d942b7847d73744abbe269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:28.524937 18131 tablet_copy_client.cc:531] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:51:28.539871 18131 tablet_bootstrap.cc:492] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap starting.
I20260810 18:51:28.549330 18140 tablet_copy_client.cc:791] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 0 data blocks...
I20260810 18:51:28.550045 18140 tablet_copy_client.cc:655] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 1 WAL segments...
I20260810 18:51:28.553860 18140 tablet_copy_client.cc:531] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:51:28.563300 18131 tablet_bootstrap.cc:492] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:51:28.564219 18131 tablet_bootstrap.cc:492] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap complete.
I20260810 18:51:28.564855 18131 ts_tablet_manager.cc:1397] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.009s
I20260810 18:51:28.567741 18140 tablet_bootstrap.cc:492] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap starting.
I20260810 18:51:28.569785 18131 raft_consensus.cc:348] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:28.570676 18131 raft_consensus.cc:729] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Initialized, Role: LEARNER
I20260810 18:51:28.571341 18131 consensus_queue.cc:260] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:28.573861 18131 ts_tablet_manager.cc:1428] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20260810 18:51:28.579540 16648 tablet_copy_service.cc:342] P 0397d760589d40d3b92ef141cdfc2dbb: Request end of tablet copy session 831925dd8b644ebab7eb0d672a94eddd-192ad39135ed48d5adab628ba706791a received from {username='slave'} at 127.0.0.1:60066
I20260810 18:51:28.579941 16648 tablet_copy_service.cc:433] P 0397d760589d40d3b92ef141cdfc2dbb: ending tablet copy session 831925dd8b644ebab7eb0d672a94eddd-192ad39135ed48d5adab628ba706791a on tablet 192ad39135ed48d5adab628ba706791a with peer 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:28.593116 18131 ts_tablet_manager.cc:927] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd: Initiating tablet copy from peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:28.598691 18131 tablet_copy_client.cc:316] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Beginning tablet copy session from remote peer at address 127.14.32.195:40171
I20260810 18:51:28.600487 16764 tablet_copy_service.cc:140] P 980f343dd1fd4e049143f8b9d9f2f1be: Received BeginTabletCopySession request for tablet 42bb15fb54124b8db9b482e3f17fc87c from peer 831925dd8b644ebab7eb0d672a94eddd ({username='slave'} at 127.0.0.1:37716)
I20260810 18:51:28.601009 16764 tablet_copy_service.cc:161] P 980f343dd1fd4e049143f8b9d9f2f1be: Beginning new tablet copy session on tablet 42bb15fb54124b8db9b482e3f17fc87c from peer 831925dd8b644ebab7eb0d672a94eddd at {username='slave'} at 127.0.0.1:37716: session id = 831925dd8b644ebab7eb0d672a94eddd-42bb15fb54124b8db9b482e3f17fc87c
I20260810 18:51:28.605614 18140 tablet_bootstrap.cc:492] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:51:28.606520 18140 tablet_bootstrap.cc:492] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap complete.
I20260810 18:51:28.607182 18140 ts_tablet_manager.cc:1397] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd: Time spent bootstrapping tablet: real 0.040s user 0.023s sys 0.005s
I20260810 18:51:28.608650 16764 tablet_copy_source_session.cc:205] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:51:28.609745 18140 raft_consensus.cc:348] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:28.612360 18140 raft_consensus.cc:729] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Initialized, Role: LEARNER
I20260810 18:51:28.611907 18131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42bb15fb54124b8db9b482e3f17fc87c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:28.614140 18140 consensus_queue.cc:260] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:28.617578 18140 ts_tablet_manager.cc:1428] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260810 18:51:28.619366 16525 tablet_copy_service.cc:342] P d304ac47fdf846de81337057a4ef7bea: Request end of tablet copy session 831925dd8b644ebab7eb0d672a94eddd-681439c754d942b7847d73744abbe269 received from {username='slave'} at 127.0.0.1:56468
I20260810 18:51:28.619771 16525 tablet_copy_service.cc:433] P d304ac47fdf846de81337057a4ef7bea: ending tablet copy session 831925dd8b644ebab7eb0d672a94eddd-681439c754d942b7847d73744abbe269 on tablet 681439c754d942b7847d73744abbe269 with peer 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:28.633545 18131 tablet_copy_client.cc:791] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 0 data blocks...
I20260810 18:51:28.634101 18131 tablet_copy_client.cc:655] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 1 WAL segments...
I20260810 18:51:28.637553 18131 tablet_copy_client.cc:531] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:51:28.645427 18131 tablet_bootstrap.cc:492] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap starting.
I20260810 18:51:28.660030 18131 tablet_bootstrap.cc:492] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:51:28.660766 18131 tablet_bootstrap.cc:492] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap complete.
I20260810 18:51:28.661311 18131 ts_tablet_manager.cc:1397] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.003s
I20260810 18:51:28.663396 18131 raft_consensus.cc:348] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:28.664157 18131 raft_consensus.cc:729] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Initialized, Role: LEARNER
I20260810 18:51:28.664583 18131 consensus_queue.cc:260] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:28.666668 18131 ts_tablet_manager.cc:1428] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260810 18:51:28.668799 16764 tablet_copy_service.cc:342] P 980f343dd1fd4e049143f8b9d9f2f1be: Request end of tablet copy session 831925dd8b644ebab7eb0d672a94eddd-42bb15fb54124b8db9b482e3f17fc87c received from {username='slave'} at 127.0.0.1:37716
I20260810 18:51:28.669186 16764 tablet_copy_service.cc:433] P 980f343dd1fd4e049143f8b9d9f2f1be: ending tablet copy session 831925dd8b644ebab7eb0d672a94eddd-42bb15fb54124b8db9b482e3f17fc87c on tablet 42bb15fb54124b8db9b482e3f17fc87c with peer 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:28.875227 17786 raft_consensus.cc:1203] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260810 18:51:28.912765 17786 raft_consensus.cc:1203] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260810 18:51:29.037308 17786 raft_consensus.cc:1203] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260810 18:51:29.092270 17786 raft_consensus.cc:1203] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260810 18:51:29.286476 18101 raft_consensus.cc:1050] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb: attempting to promote NON_VOTER 831925dd8b644ebab7eb0d672a94eddd to VOTER
I20260810 18:51:29.288869 18101 consensus_queue.cc:237] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:29.296018 17786 raft_consensus.cc:1261] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:29.297636 16501 raft_consensus.cc:1261] T abdb4e808ce3454ab9a52051b369e5d0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:29.298655 16743 raft_consensus.cc:1261] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:29.298707 18101 consensus_queue.cc:1035] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:29.299926 18008 consensus_queue.cc:1035] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:29.300796 18105 consensus_queue.cc:1035] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:29.352802 18008 raft_consensus.cc:2932] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.355430 17786 raft_consensus.cc:2932] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.362442 16501 raft_consensus.cc:2932] T abdb4e808ce3454ab9a52051b369e5d0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.364877 16749 raft_consensus.cc:2932] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.370944 16343 catalog_manager.cc:5495] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:51:29.425639 17924 raft_consensus.cc:1050] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea: attempting to promote NON_VOTER 831925dd8b644ebab7eb0d672a94eddd to VOTER
I20260810 18:51:29.428035 17924 consensus_queue.cc:237] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:29.424480 16631 consensus_queue.cc:237] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:29.445367 17786 raft_consensus.cc:1261] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:29.445652 17785 raft_consensus.cc:1261] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:29.446627 16631 raft_consensus.cc:1261] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:29.447382 16749 raft_consensus.cc:1261] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:29.447384 18008 consensus_queue.cc:1035] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:51:29.449147 17924 consensus_queue.cc:1035] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:29.450181 17977 consensus_queue.cc:1035] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:29.450671 16749 raft_consensus.cc:1261] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:29.451505 18123 consensus_queue.cc:1035] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:29.454038 18173 consensus_queue.cc:1035] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260810 18:51:29.459362 18008 raft_consensus.cc:2932] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.461894 16743 raft_consensus.cc:2932] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.466091 17785 raft_consensus.cc:2932] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.481505 17977 raft_consensus.cc:2932] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.485956 16324 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet abdb4e808ce3454ab9a52051b369e5d0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 18:51:29.486285 16344 catalog_manager.cc:5495] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.492383 16749 raft_consensus.cc:2932] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.496649 17786 raft_consensus.cc:2932] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.509788 16631 raft_consensus.cc:2932] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.520413 16492 tablet_service.cc:1514] Processing DeleteTablet for tablet abdb4e808ce3454ab9a52051b369e5d0 with delete_type TABLET_DATA_TOMBSTONED (TS d304ac47fdf846de81337057a4ef7bea not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52330
I20260810 18:51:29.525610 16342 catalog_manager.cc:5495] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.590143 18184 tablet_replica.cc:330] T abdb4e808ce3454ab9a52051b369e5d0 P d304ac47fdf846de81337057a4ef7bea: stopping tablet replica
I20260810 18:51:29.591329 18184 raft_consensus.cc:2229] T abdb4e808ce3454ab9a52051b369e5d0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:29.592070 18184 raft_consensus.cc:2258] T abdb4e808ce3454ab9a52051b369e5d0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:29.595600 18184 ts_tablet_manager.cc:1905] T abdb4e808ce3454ab9a52051b369e5d0 P d304ac47fdf846de81337057a4ef7bea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 18:51:29.611537 18184 ts_tablet_manager.cc:1918] T abdb4e808ce3454ab9a52051b369e5d0 P d304ac47fdf846de81337057a4ef7bea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260810 18:51:29.611915 18184 log.cc:1196] T abdb4e808ce3454ab9a52051b369e5d0 P d304ac47fdf846de81337057a4ef7bea: Deleting WAL directory at /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-0-root/wals/abdb4e808ce3454ab9a52051b369e5d0
I20260810 18:51:29.612305 18101 raft_consensus.cc:1050] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb: attempting to promote NON_VOTER 831925dd8b644ebab7eb0d672a94eddd to VOTER
I20260810 18:51:29.614753 18101 consensus_queue.cc:237] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:29.614108 16323 catalog_manager.cc:4841] TS d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): tablet abdb4e808ce3454ab9a52051b369e5d0 (table test-workload [id=5aa4952cbee8447ca410249e65af5af6]) successfully deleted
I20260810 18:51:29.620257 17786 raft_consensus.cc:1261] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:29.621568 18105 consensus_queue.cc:1035] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:29.629843 16501 raft_consensus.cc:1261] T 192ad39135ed48d5adab628ba706791a P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:29.636484 18008 consensus_queue.cc:1035] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:29.642087 16743 raft_consensus.cc:1261] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:29.652341 18191 raft_consensus.cc:1050] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be: attempting to promote NON_VOTER 831925dd8b644ebab7eb0d672a94eddd to VOTER
I20260810 18:51:29.656038 16506 consensus_queue.cc:237] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:29.656068 18191 consensus_queue.cc:237] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:29.656622 18173 raft_consensus.cc:2932] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.663553 17786 raft_consensus.cc:1261] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:29.665092 17786 raft_consensus.cc:1261] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:29.666090 16631 raft_consensus.cc:1261] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:29.666814 18086 consensus_queue.cc:1035] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:29.667333 17977 consensus_queue.cc:1035] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:51:29.668632 18183 consensus_queue.cc:1035] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:51:29.673871 18101 consensus_queue.cc:1035] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:29.668154 16501 raft_consensus.cc:2932] T 192ad39135ed48d5adab628ba706791a P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.682488 17785 raft_consensus.cc:2932] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.683116 16631 raft_consensus.cc:2932] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.679107 18183 raft_consensus.cc:2932] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.681982 17786 raft_consensus.cc:2932] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.700271 16343 catalog_manager.cc:5495] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
W20260810 18:51:29.717823 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 44713.9us
I20260810 18:51:29.720501 16343 catalog_manager.cc:5495] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: config changed from index 3 to 4, VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) evicted. New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.731251 16726 tablet_service.cc:1514] Processing DeleteTablet for tablet 681439c754d942b7847d73744abbe269 with delete_type TABLET_DATA_TOMBSTONED (TS 980f343dd1fd4e049143f8b9d9f2f1be not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43720
I20260810 18:51:29.736035 16323 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 681439c754d942b7847d73744abbe269 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 18:51:29.741753 18198 tablet_replica.cc:330] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be: stopping tablet replica
I20260810 18:51:29.742810 16749 raft_consensus.cc:2932] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.745278 18198 raft_consensus.cc:2229] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:29.746225 18198 raft_consensus.cc:2258] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:29.764009 18198 ts_tablet_manager.cc:1905] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 18:51:29.771648 16505 raft_consensus.cc:1261] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:29.773553 18086 consensus_queue.cc:1035] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:29.787607 16631 raft_consensus.cc:1261] T 42bb15fb54124b8db9b482e3f17fc87c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:29.789887 18198 ts_tablet_manager.cc:1918] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260810 18:51:29.790403 18198 log.cc:1196] T 681439c754d942b7847d73744abbe269 P 980f343dd1fd4e049143f8b9d9f2f1be: Deleting WAL directory at /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-2-root/wals/681439c754d942b7847d73744abbe269
I20260810 18:51:29.791981 16328 catalog_manager.cc:4841] TS 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): tablet 681439c754d942b7847d73744abbe269 (table test-workload [id=5aa4952cbee8447ca410249e65af5af6]) successfully deleted
I20260810 18:51:29.792647 18114 consensus_queue.cc:1035] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:29.794240 18086 raft_consensus.cc:2932] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.810523 16505 raft_consensus.cc:2932] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.817646 17785 raft_consensus.cc:2932] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.829432 16342 catalog_manager.cc:5495] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea reported cstate change: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.830964 16623 raft_consensus.cc:2932] T 42bb15fb54124b8db9b482e3f17fc87c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.862330 16749 consensus_queue.cc:237] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:29.868804 17785 raft_consensus.cc:1261] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:29.869517 16506 raft_consensus.cc:1261] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:29.871778 18086 consensus_queue.cc:1035] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:51:29.872665 18205 consensus_queue.cc:1035] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:51:29.882992 18205 raft_consensus.cc:2932] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.888154 16506 raft_consensus.cc:2932] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.899778 17787 raft_consensus.cc:2932] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.912143 16342 catalog_manager.cc:5495] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea reported cstate change: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:29.914343 16328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 42bb15fb54124b8db9b482e3f17fc87c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 18:51:29.938546 16605 tablet_service.cc:1514] Processing DeleteTablet for tablet 42bb15fb54124b8db9b482e3f17fc87c with delete_type TABLET_DATA_TOMBSTONED (TS 0397d760589d40d3b92ef141cdfc2dbb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53868
I20260810 18:51:29.941275 18208 tablet_replica.cc:330] T 42bb15fb54124b8db9b482e3f17fc87c P 0397d760589d40d3b92ef141cdfc2dbb: stopping tablet replica
I20260810 18:51:29.950479 18208 raft_consensus.cc:2229] T 42bb15fb54124b8db9b482e3f17fc87c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:29.951421 18208 raft_consensus.cc:2258] T 42bb15fb54124b8db9b482e3f17fc87c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:29.956257 18208 ts_tablet_manager.cc:1905] T 42bb15fb54124b8db9b482e3f17fc87c P 0397d760589d40d3b92ef141cdfc2dbb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 18:51:29.981208 18208 ts_tablet_manager.cc:1918] T 42bb15fb54124b8db9b482e3f17fc87c P 0397d760589d40d3b92ef141cdfc2dbb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260810 18:51:29.981882 18208 log.cc:1196] T 42bb15fb54124b8db9b482e3f17fc87c P 0397d760589d40d3b92ef141cdfc2dbb: Deleting WAL directory at /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-1-root/wals/42bb15fb54124b8db9b482e3f17fc87c
I20260810 18:51:29.984828 16324 catalog_manager.cc:4841] TS 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): tablet 42bb15fb54124b8db9b482e3f17fc87c (table test-workload [id=5aa4952cbee8447ca410249e65af5af6]) successfully deleted
I20260810 18:51:30.108788 16631 consensus_queue.cc:237] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:30.115839 16749 raft_consensus.cc:1261] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:30.117503 18173 consensus_queue.cc:1035] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:51:30.118764 17787 raft_consensus.cc:1261] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:30.120267 18173 consensus_queue.cc:1035] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:51:30.126997 18105 raft_consensus.cc:2932] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:30.129868 17787 raft_consensus.cc:2932] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:30.132220 16749 raft_consensus.cc:2932] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:30.143002 16324 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 192ad39135ed48d5adab628ba706791a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 18:51:30.153369 16342 catalog_manager.cc:5495] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:30.172312 16492 tablet_service.cc:1514] Processing DeleteTablet for tablet 192ad39135ed48d5adab628ba706791a with delete_type TABLET_DATA_TOMBSTONED (TS d304ac47fdf846de81337057a4ef7bea not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52330
I20260810 18:51:30.174795 18209 tablet_replica.cc:330] T 192ad39135ed48d5adab628ba706791a P d304ac47fdf846de81337057a4ef7bea: stopping tablet replica
I20260810 18:51:30.176105 18209 raft_consensus.cc:2229] T 192ad39135ed48d5adab628ba706791a P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:30.176877 18209 raft_consensus.cc:2258] T 192ad39135ed48d5adab628ba706791a P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:30.181313 18209 ts_tablet_manager.cc:1905] T 192ad39135ed48d5adab628ba706791a P d304ac47fdf846de81337057a4ef7bea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 18:51:30.198338 18209 ts_tablet_manager.cc:1918] T 192ad39135ed48d5adab628ba706791a P d304ac47fdf846de81337057a4ef7bea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260810 18:51:30.198815 18209 log.cc:1196] T 192ad39135ed48d5adab628ba706791a P d304ac47fdf846de81337057a4ef7bea: Deleting WAL directory at /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-0-root/wals/192ad39135ed48d5adab628ba706791a
I20260810 18:51:30.205019 16323 catalog_manager.cc:4841] TS d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): tablet 192ad39135ed48d5adab628ba706791a (table test-workload [id=5aa4952cbee8447ca410249e65af5af6]) successfully deleted
I20260810 18:51:31.210363 16506 consensus_queue.cc:237] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
W20260810 18:51:31.216920 16447 consensus_peers.cc:487] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea -> Peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387): Couldn't send request to peer 831925dd8b644ebab7eb0d672a94eddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b4248aab7d8b4ed0b7c298cad63430f8. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:31.217545 16749 raft_consensus.cc:1261] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:31.219964 18183 consensus_queue.cc:1035] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:31.221156 16631 raft_consensus.cc:1261] T b4248aab7d8b4ed0b7c298cad63430f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:31.222365 18183 consensus_queue.cc:1035] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:31.227487 18123 raft_consensus.cc:2932] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:31.229722 16749 raft_consensus.cc:2932] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:31.243283 16623 raft_consensus.cc:2932] T b4248aab7d8b4ed0b7c298cad63430f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:31.246089 16631 consensus_queue.cc:237] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:31.251928 16343 catalog_manager.cc:5495] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea reported cstate change: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260810 18:51:31.265126 16573 consensus_peers.cc:487] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb -> Peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387): Couldn't send request to peer 831925dd8b644ebab7eb0d672a94eddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f3efea4006f43ee933f50d3e328a1f0. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:31.272886 16506 raft_consensus.cc:1261] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:31.274439 18105 consensus_queue.cc:1035] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:31.275673 16749 raft_consensus.cc:1261] T 3f3efea4006f43ee933f50d3e328a1f0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:31.278007 18105 consensus_queue.cc:1035] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:31.301789 18223 raft_consensus.cc:2932] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:31.304064 16749 raft_consensus.cc:2932] T 3f3efea4006f43ee933f50d3e328a1f0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:31.311702 16506 raft_consensus.cc:2932] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:31.334069 16631 consensus_queue.cc:237] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:31.339902 16342 catalog_manager.cc:5495] T 3f3efea4006f43ee933f50d3e328a1f0 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
W20260810 18:51:31.357872 16573 consensus_peers.cc:487] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb -> Peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387): Couldn't send request to peer 831925dd8b644ebab7eb0d672a94eddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e58f8d2231b4e2d900ec099ae700d09. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:31.364044 16746 raft_consensus.cc:1261] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:31.365363 16506 raft_consensus.cc:1261] T 3e58f8d2231b4e2d900ec099ae700d09 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:31.367326 18223 consensus_queue.cc:1035] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:31.368403 18222 consensus_queue.cc:1035] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:31.378646 18223 raft_consensus.cc:2932] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:31.381639 16749 raft_consensus.cc:2932] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:31.387259 16501 raft_consensus.cc:2932] T 3e58f8d2231b4e2d900ec099ae700d09 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:31.393769 16749 consensus_queue.cc:237] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:31.400615 16343 catalog_manager.cc:5495] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:31.416625 16636 raft_consensus.cc:1261] T 649d43610ba24875a4418e39b25126ba P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260810 18:51:31.416875 16695 consensus_peers.cc:487] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be -> Peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387): Couldn't send request to peer 831925dd8b644ebab7eb0d672a94eddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 649d43610ba24875a4418e39b25126ba. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:31.417841 18232 consensus_queue.cc:1035] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:31.418157 16506 raft_consensus.cc:1261] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:31.419510 18205 consensus_queue.cc:1035] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:31.425937 18232 raft_consensus.cc:2932] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:31.429934 16501 raft_consensus.cc:2932] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:31.431288 16636 raft_consensus.cc:2932] T 649d43610ba24875a4418e39b25126ba P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:31.447151 16344 catalog_manager.cc:5495] T 649d43610ba24875a4418e39b25126ba P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:31.649358 18242 ts_tablet_manager.cc:927] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd: Initiating tablet copy from peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791)
I20260810 18:51:31.651145 18242 tablet_copy_client.cc:316] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Beginning tablet copy session from remote peer at address 127.14.32.193:40791
I20260810 18:51:31.652585 16525 tablet_copy_service.cc:140] P d304ac47fdf846de81337057a4ef7bea: Received BeginTabletCopySession request for tablet b4248aab7d8b4ed0b7c298cad63430f8 from peer 831925dd8b644ebab7eb0d672a94eddd ({username='slave'} at 127.0.0.1:56468)
I20260810 18:51:31.653091 16525 tablet_copy_service.cc:161] P d304ac47fdf846de81337057a4ef7bea: Beginning new tablet copy session on tablet b4248aab7d8b4ed0b7c298cad63430f8 from peer 831925dd8b644ebab7eb0d672a94eddd at {username='slave'} at 127.0.0.1:56468: session id = 831925dd8b644ebab7eb0d672a94eddd-b4248aab7d8b4ed0b7c298cad63430f8
I20260810 18:51:31.660069 16525 tablet_copy_source_session.cc:205] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:51:31.663313 18242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4248aab7d8b4ed0b7c298cad63430f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:31.675835 18242 tablet_copy_client.cc:791] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 0 data blocks...
I20260810 18:51:31.676378 18242 tablet_copy_client.cc:655] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 1 WAL segments...
I20260810 18:51:31.679590 18242 tablet_copy_client.cc:531] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:51:31.691771 18242 tablet_bootstrap.cc:492] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap starting.
I20260810 18:51:31.719959 18242 tablet_bootstrap.cc:492] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:51:31.720718 18242 tablet_bootstrap.cc:492] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap complete.
I20260810 18:51:31.721320 18242 ts_tablet_manager.cc:1397] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.010s
I20260810 18:51:31.723743 18242 raft_consensus.cc:348] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:31.724587 18242 raft_consensus.cc:729] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Initialized, Role: LEARNER
I20260810 18:51:31.725194 18242 consensus_queue.cc:260] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:31.734218 18242 ts_tablet_manager.cc:1428] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd: Time spent starting tablet: real 0.013s user 0.006s sys 0.005s
I20260810 18:51:31.735822 16525 tablet_copy_service.cc:342] P d304ac47fdf846de81337057a4ef7bea: Request end of tablet copy session 831925dd8b644ebab7eb0d672a94eddd-b4248aab7d8b4ed0b7c298cad63430f8 received from {username='slave'} at 127.0.0.1:56468
I20260810 18:51:31.736124 16525 tablet_copy_service.cc:433] P d304ac47fdf846de81337057a4ef7bea: ending tablet copy session 831925dd8b644ebab7eb0d672a94eddd-b4248aab7d8b4ed0b7c298cad63430f8 on tablet b4248aab7d8b4ed0b7c298cad63430f8 with peer 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:31.738127 18242 ts_tablet_manager.cc:927] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd: Initiating tablet copy from peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:31.747057 18247 ts_tablet_manager.cc:927] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd: Initiating tablet copy from peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:31.747795 18242 tablet_copy_client.cc:316] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Beginning tablet copy session from remote peer at address 127.14.32.194:45827
I20260810 18:51:31.749301 16648 tablet_copy_service.cc:140] P 0397d760589d40d3b92ef141cdfc2dbb: Received BeginTabletCopySession request for tablet 3f3efea4006f43ee933f50d3e328a1f0 from peer 831925dd8b644ebab7eb0d672a94eddd ({username='slave'} at 127.0.0.1:60066)
I20260810 18:51:31.749835 16648 tablet_copy_service.cc:161] P 0397d760589d40d3b92ef141cdfc2dbb: Beginning new tablet copy session on tablet 3f3efea4006f43ee933f50d3e328a1f0 from peer 831925dd8b644ebab7eb0d672a94eddd at {username='slave'} at 127.0.0.1:60066: session id = 831925dd8b644ebab7eb0d672a94eddd-3f3efea4006f43ee933f50d3e328a1f0
I20260810 18:51:31.757305 16648 tablet_copy_source_session.cc:205] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:51:31.759313 18247 tablet_copy_client.cc:316] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Beginning tablet copy session from remote peer at address 127.14.32.194:45827
I20260810 18:51:31.761569 18242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f3efea4006f43ee933f50d3e328a1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:31.768175 16648 tablet_copy_service.cc:140] P 0397d760589d40d3b92ef141cdfc2dbb: Received BeginTabletCopySession request for tablet 3e58f8d2231b4e2d900ec099ae700d09 from peer 831925dd8b644ebab7eb0d672a94eddd ({username='slave'} at 127.0.0.1:60066)
I20260810 18:51:31.768688 16648 tablet_copy_service.cc:161] P 0397d760589d40d3b92ef141cdfc2dbb: Beginning new tablet copy session on tablet 3e58f8d2231b4e2d900ec099ae700d09 from peer 831925dd8b644ebab7eb0d672a94eddd at {username='slave'} at 127.0.0.1:60066: session id = 831925dd8b644ebab7eb0d672a94eddd-3e58f8d2231b4e2d900ec099ae700d09
I20260810 18:51:31.776028 16648 tablet_copy_source_session.cc:205] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:51:31.776961 18242 tablet_copy_client.cc:791] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 0 data blocks...
I20260810 18:51:31.777561 18242 tablet_copy_client.cc:655] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 1 WAL segments...
I20260810 18:51:31.779650 18247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e58f8d2231b4e2d900ec099ae700d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:31.786468 18242 tablet_copy_client.cc:531] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:51:31.794466 18242 tablet_bootstrap.cc:492] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap starting.
I20260810 18:51:31.795226 18247 tablet_copy_client.cc:791] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 0 data blocks...
I20260810 18:51:31.795784 18247 tablet_copy_client.cc:655] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 1 WAL segments...
I20260810 18:51:31.802969 18247 tablet_copy_client.cc:531] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:51:31.810384 18247 tablet_bootstrap.cc:492] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap starting.
I20260810 18:51:31.840163 18242 tablet_bootstrap.cc:492] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:51:31.841146 18242 tablet_bootstrap.cc:492] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap complete.
I20260810 18:51:31.841928 18242 ts_tablet_manager.cc:1397] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd: Time spent bootstrapping tablet: real 0.048s user 0.018s sys 0.004s
I20260810 18:51:31.847564 18242 raft_consensus.cc:348] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:31.848796 18242 raft_consensus.cc:729] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Initialized, Role: LEARNER
I20260810 18:51:31.849695 18242 consensus_queue.cc:260] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:31.853363 18242 ts_tablet_manager.cc:1428] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd: Time spent starting tablet: real 0.011s user 0.009s sys 0.003s
I20260810 18:51:31.857273 16648 tablet_copy_service.cc:342] P 0397d760589d40d3b92ef141cdfc2dbb: Request end of tablet copy session 831925dd8b644ebab7eb0d672a94eddd-3f3efea4006f43ee933f50d3e328a1f0 received from {username='slave'} at 127.0.0.1:60066
I20260810 18:51:31.857847 16648 tablet_copy_service.cc:433] P 0397d760589d40d3b92ef141cdfc2dbb: ending tablet copy session 831925dd8b644ebab7eb0d672a94eddd-3f3efea4006f43ee933f50d3e328a1f0 on tablet 3f3efea4006f43ee933f50d3e328a1f0 with peer 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:31.862779 18247 tablet_bootstrap.cc:492] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:51:31.864074 18247 tablet_bootstrap.cc:492] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap complete.
I20260810 18:51:31.865054 18247 ts_tablet_manager.cc:1397] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd: Time spent bootstrapping tablet: real 0.055s user 0.024s sys 0.025s
I20260810 18:51:31.871305 18247 raft_consensus.cc:348] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:31.872128 18247 raft_consensus.cc:729] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Initialized, Role: LEARNER
I20260810 18:51:31.872802 18247 consensus_queue.cc:260] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:31.876304 18247 ts_tablet_manager.cc:1428] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s
I20260810 18:51:31.879784 16648 tablet_copy_service.cc:342] P 0397d760589d40d3b92ef141cdfc2dbb: Request end of tablet copy session 831925dd8b644ebab7eb0d672a94eddd-3e58f8d2231b4e2d900ec099ae700d09 received from {username='slave'} at 127.0.0.1:60066
I20260810 18:51:31.880483 16648 tablet_copy_service.cc:433] P 0397d760589d40d3b92ef141cdfc2dbb: ending tablet copy session 831925dd8b644ebab7eb0d672a94eddd-3e58f8d2231b4e2d900ec099ae700d09 on tablet 3e58f8d2231b4e2d900ec099ae700d09 with peer 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:31.891462 18247 ts_tablet_manager.cc:927] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd: Initiating tablet copy from peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:31.894893 18247 tablet_copy_client.cc:316] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Beginning tablet copy session from remote peer at address 127.14.32.195:40171
I20260810 18:51:31.896545 16764 tablet_copy_service.cc:140] P 980f343dd1fd4e049143f8b9d9f2f1be: Received BeginTabletCopySession request for tablet 649d43610ba24875a4418e39b25126ba from peer 831925dd8b644ebab7eb0d672a94eddd ({username='slave'} at 127.0.0.1:37716)
I20260810 18:51:31.897137 16764 tablet_copy_service.cc:161] P 980f343dd1fd4e049143f8b9d9f2f1be: Beginning new tablet copy session on tablet 649d43610ba24875a4418e39b25126ba from peer 831925dd8b644ebab7eb0d672a94eddd at {username='slave'} at 127.0.0.1:37716: session id = 831925dd8b644ebab7eb0d672a94eddd-649d43610ba24875a4418e39b25126ba
I20260810 18:51:31.908052 16764 tablet_copy_source_session.cc:205] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:51:31.912318 18247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 649d43610ba24875a4418e39b25126ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:31.932394 18247 tablet_copy_client.cc:791] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 0 data blocks...
I20260810 18:51:31.933128 18247 tablet_copy_client.cc:655] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 1 WAL segments...
I20260810 18:51:31.940940 18247 tablet_copy_client.cc:531] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:51:31.952211 18247 tablet_bootstrap.cc:492] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap starting.
I20260810 18:51:31.974112 18247 tablet_bootstrap.cc:492] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:51:31.975003 18247 tablet_bootstrap.cc:492] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap complete.
I20260810 18:51:31.975677 18247 ts_tablet_manager.cc:1397] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd: Time spent bootstrapping tablet: real 0.025s user 0.015s sys 0.009s
I20260810 18:51:31.978194 18247 raft_consensus.cc:348] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:31.978927 18247 raft_consensus.cc:729] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Initialized, Role: LEARNER
I20260810 18:51:31.979565 18247 consensus_queue.cc:260] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:31.981731 18247 ts_tablet_manager.cc:1428] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:51:31.984620 16764 tablet_copy_service.cc:342] P 980f343dd1fd4e049143f8b9d9f2f1be: Request end of tablet copy session 831925dd8b644ebab7eb0d672a94eddd-649d43610ba24875a4418e39b25126ba received from {username='slave'} at 127.0.0.1:37716
I20260810 18:51:31.984987 16764 tablet_copy_service.cc:433] P 980f343dd1fd4e049143f8b9d9f2f1be: ending tablet copy session 831925dd8b644ebab7eb0d672a94eddd-649d43610ba24875a4418e39b25126ba on tablet 649d43610ba24875a4418e39b25126ba with peer 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:32.186393 17787 raft_consensus.cc:1203] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260810 18:51:32.210942 17787 raft_consensus.cc:1203] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260810 18:51:32.246537 17787 raft_consensus.cc:1203] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260810 18:51:32.461283 17787 raft_consensus.cc:1203] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260810 18:51:32.639415 18123 raft_consensus.cc:1050] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea: attempting to promote NON_VOTER 831925dd8b644ebab7eb0d672a94eddd to VOTER
I20260810 18:51:32.641379 18123 consensus_queue.cc:237] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:32.648530 17787 raft_consensus.cc:1261] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:32.649412 16622 raft_consensus.cc:1261] T b4248aab7d8b4ed0b7c298cad63430f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:32.648495 16749 raft_consensus.cc:1261] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:32.650306 18214 consensus_queue.cc:1035] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:32.651914 18270 consensus_queue.cc:1035] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:32.653070 18183 consensus_queue.cc:1035] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:32.667440 18223 raft_consensus.cc:1050] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb: attempting to promote NON_VOTER 831925dd8b644ebab7eb0d672a94eddd to VOTER
I20260810 18:51:32.671774 18183 raft_consensus.cc:2932] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:32.673821 17785 raft_consensus.cc:2932] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:32.672473 18223 consensus_queue.cc:237] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:32.685408 18222 raft_consensus.cc:1050] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb: attempting to promote NON_VOTER 831925dd8b644ebab7eb0d672a94eddd to VOTER
I20260810 18:51:32.687961 18222 consensus_queue.cc:237] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:32.682653 16631 raft_consensus.cc:2932] T b4248aab7d8b4ed0b7c298cad63430f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:32.701869 16343 catalog_manager.cc:5495] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:32.703770 16750 raft_consensus.cc:2932] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:32.717013 17785 raft_consensus.cc:1261] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:32.719766 18008 consensus_queue.cc:1035] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:32.726138 16743 raft_consensus.cc:1261] T 3f3efea4006f43ee933f50d3e328a1f0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:32.740070 18260 consensus_queue.cc:1035] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:32.750560 16749 raft_consensus.cc:1261] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:32.753540 18008 consensus_queue.cc:1035] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:32.791416 17783 raft_consensus.cc:1261] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:32.796319 18222 consensus_queue.cc:1035] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:32.797574 16507 raft_consensus.cc:1261] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:32.797703 16505 raft_consensus.cc:1261] T 3e58f8d2231b4e2d900ec099ae700d09 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:32.800232 18290 consensus_queue.cc:1035] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:32.802459 18290 consensus_queue.cc:1035] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:32.812184 18288 raft_consensus.cc:2932] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:32.816550 16743 raft_consensus.cc:2932] T 3f3efea4006f43ee933f50d3e328a1f0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:32.824846 18290 raft_consensus.cc:2932] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:32.827597 16506 raft_consensus.cc:2932] T 3e58f8d2231b4e2d900ec099ae700d09 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:32.842499 17783 raft_consensus.cc:2932] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:32.845966 16501 raft_consensus.cc:2932] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:32.875524 16501 consensus_queue.cc:237] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:32.881485 17785 raft_consensus.cc:2932] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:32.882498 16344 catalog_manager.cc:5495] T 3f3efea4006f43ee933f50d3e328a1f0 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:32.891637 17783 raft_consensus.cc:1261] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:32.894840 18123 consensus_queue.cc:1035] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:51:32.895354 16343 catalog_manager.cc:5495] T 3e58f8d2231b4e2d900ec099ae700d09 P d304ac47fdf846de81337057a4ef7bea reported cstate change: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:32.899523 16750 raft_consensus.cc:2932] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:32.906121 18232 raft_consensus.cc:1050] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be: attempting to promote NON_VOTER 831925dd8b644ebab7eb0d672a94eddd to VOTER
I20260810 18:51:32.903123 16743 raft_consensus.cc:1261] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:32.910166 18123 consensus_queue.cc:1035] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:51:32.910284 18232 consensus_queue.cc:237] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:32.913611 18183 raft_consensus.cc:2932] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:32.922410 16746 raft_consensus.cc:2932] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:32.941547 16631 consensus_queue.cc:237] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:32.953042 16323 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet b4248aab7d8b4ed0b7c298cad63430f8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 18:51:32.962435 17786 raft_consensus.cc:1261] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:32.964867 16501 raft_consensus.cc:1261] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:32.976390 18301 consensus_queue.cc:1035] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:51:32.964184 17783 raft_consensus.cc:2932] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:32.978335 16636 consensus_queue.cc:237] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:32.989429 18223 consensus_queue.cc:1035] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:51:32.990759 17786 raft_consensus.cc:1261] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:32.995047 16750 raft_consensus.cc:1261] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:33.001482 18223 consensus_queue.cc:1035] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260810 18:51:33.008098 16343 catalog_manager.cc:5495] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea reported cstate change: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:51:33.018587 16506 raft_consensus.cc:2932] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:33.015097 18288 raft_consensus.cc:2932] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:33.034106 16605 tablet_service.cc:1514] Processing DeleteTablet for tablet b4248aab7d8b4ed0b7c298cad63430f8 with delete_type TABLET_DATA_TOMBSTONED (TS 0397d760589d40d3b92ef141cdfc2dbb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53868
I20260810 18:51:33.036217 17785 raft_consensus.cc:2932] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:33.060253 18301 raft_consensus.cc:2932] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:33.067823 17786 raft_consensus.cc:2932] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:33.094587 16343 catalog_manager.cc:5495] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea reported cstate change: config changed from index 3 to 4, VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) evicted. New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:33.098340 16344 catalog_manager.cc:5495] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:33.110531 16726 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f3efea4006f43ee933f50d3e328a1f0 with delete_type TABLET_DATA_TOMBSTONED (TS 980f343dd1fd4e049143f8b9d9f2f1be not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43720
I20260810 18:51:33.111472 16324 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 3e58f8d2231b4e2d900ec099ae700d09 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 18:51:33.121254 16492 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e58f8d2231b4e2d900ec099ae700d09 with delete_type TABLET_DATA_TOMBSTONED (TS d304ac47fdf846de81337057a4ef7bea not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52330
I20260810 18:51:33.150892 17785 raft_consensus.cc:1261] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:33.152045 18319 tablet_replica.cc:330] T b4248aab7d8b4ed0b7c298cad63430f8 P 0397d760589d40d3b92ef141cdfc2dbb: stopping tablet replica
I20260810 18:51:33.158651 16506 raft_consensus.cc:1261] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:33.154955 18264 consensus_queue.cc:1035] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20260810 18:51:33.180907 18310 consensus_queue.cc:1035] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:33.211036 18324 tablet_replica.cc:330] T 3e58f8d2231b4e2d900ec099ae700d09 P d304ac47fdf846de81337057a4ef7bea: stopping tablet replica
I20260810 18:51:33.228046 18324 raft_consensus.cc:2229] T 3e58f8d2231b4e2d900ec099ae700d09 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:33.228915 18324 raft_consensus.cc:2258] T 3e58f8d2231b4e2d900ec099ae700d09 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:33.235553 18319 raft_consensus.cc:2229] T b4248aab7d8b4ed0b7c298cad63430f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:33.237489 18319 raft_consensus.cc:2258] T b4248aab7d8b4ed0b7c298cad63430f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 18:51:33.238363 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 166571us
I20260810 18:51:33.239079 18322 tablet_replica.cc:330] T 3f3efea4006f43ee933f50d3e328a1f0 P 980f343dd1fd4e049143f8b9d9f2f1be: stopping tablet replica
I20260810 18:51:33.243830 18319 ts_tablet_manager.cc:1905] T b4248aab7d8b4ed0b7c298cad63430f8 P 0397d760589d40d3b92ef141cdfc2dbb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 18:51:33.253021 18290 consensus_queue.cc:1035] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:51:33.260164 16622 raft_consensus.cc:1261] T 649d43610ba24875a4418e39b25126ba P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:33.265184 18322 raft_consensus.cc:2229] T 3f3efea4006f43ee933f50d3e328a1f0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:33.266871 18324 ts_tablet_manager.cc:1905] T 3e58f8d2231b4e2d900ec099ae700d09 P d304ac47fdf846de81337057a4ef7bea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 18:51:33.274837 16324 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 3f3efea4006f43ee933f50d3e328a1f0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 18:51:33.266283 18322 raft_consensus.cc:2258] T 3f3efea4006f43ee933f50d3e328a1f0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:33.280937 17785 raft_consensus.cc:2932] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:33.278321 18310 raft_consensus.cc:2932] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:33.290380 18322 ts_tablet_manager.cc:1905] T 3f3efea4006f43ee933f50d3e328a1f0 P 980f343dd1fd4e049143f8b9d9f2f1be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 18:51:33.304035 16743 raft_consensus.cc:2932] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:33.304301 16344 catalog_manager.cc:5495] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:33.329566 16506 raft_consensus.cc:2932] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:33.333329 18319 ts_tablet_manager.cc:1918] T b4248aab7d8b4ed0b7c298cad63430f8 P 0397d760589d40d3b92ef141cdfc2dbb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260810 18:51:33.333861 18319 log.cc:1196] T b4248aab7d8b4ed0b7c298cad63430f8 P 0397d760589d40d3b92ef141cdfc2dbb: Deleting WAL directory at /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-1-root/wals/b4248aab7d8b4ed0b7c298cad63430f8
I20260810 18:51:33.340278 16324 catalog_manager.cc:4841] TS 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): tablet b4248aab7d8b4ed0b7c298cad63430f8 (table test-workload [id=5aa4952cbee8447ca410249e65af5af6]) successfully deleted
I20260810 18:51:33.344734 18322 ts_tablet_manager.cc:1918] T 3f3efea4006f43ee933f50d3e328a1f0 P 980f343dd1fd4e049143f8b9d9f2f1be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260810 18:51:33.345299 18322 log.cc:1196] T 3f3efea4006f43ee933f50d3e328a1f0 P 980f343dd1fd4e049143f8b9d9f2f1be: Deleting WAL directory at /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-2-root/wals/3f3efea4006f43ee933f50d3e328a1f0
I20260810 18:51:33.344753 18324 ts_tablet_manager.cc:1918] T 3e58f8d2231b4e2d900ec099ae700d09 P d304ac47fdf846de81337057a4ef7bea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260810 18:51:33.347632 18324 log.cc:1196] T 3e58f8d2231b4e2d900ec099ae700d09 P d304ac47fdf846de81337057a4ef7bea: Deleting WAL directory at /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-0-root/wals/3e58f8d2231b4e2d900ec099ae700d09
I20260810 18:51:33.349684 16323 catalog_manager.cc:4841] TS d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): tablet 3e58f8d2231b4e2d900ec099ae700d09 (table test-workload [id=5aa4952cbee8447ca410249e65af5af6]) successfully deleted
I20260810 18:51:33.349880 16328 catalog_manager.cc:4841] TS 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): tablet 3f3efea4006f43ee933f50d3e328a1f0 (table test-workload [id=5aa4952cbee8447ca410249e65af5af6]) successfully deleted
I20260810 18:51:33.366195 18264 consensus_queue.cc:1035] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:33.379384 16636 raft_consensus.cc:2932] T 649d43610ba24875a4418e39b25126ba P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:33.392213 16743 consensus_queue.cc:237] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:33.399078 16506 raft_consensus.cc:1261] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:33.400746 17785 raft_consensus.cc:1261] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:33.401890 18205 consensus_queue.cc:1035] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:51:33.402701 18264 consensus_queue.cc:1035] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:51:33.412153 18205 raft_consensus.cc:2932] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:33.419791 17785 raft_consensus.cc:2932] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:33.420573 16506 raft_consensus.cc:2932] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:33.425395 16328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 649d43610ba24875a4418e39b25126ba with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 18:51:33.437453 16344 catalog_manager.cc:5495] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:51:33.457067 16605 tablet_service.cc:1514] Processing DeleteTablet for tablet 649d43610ba24875a4418e39b25126ba with delete_type TABLET_DATA_TOMBSTONED (TS 0397d760589d40d3b92ef141cdfc2dbb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53868
I20260810 18:51:33.458636 18319 tablet_replica.cc:330] T 649d43610ba24875a4418e39b25126ba P 0397d760589d40d3b92ef141cdfc2dbb: stopping tablet replica
I20260810 18:51:33.460740 18319 raft_consensus.cc:2229] T 649d43610ba24875a4418e39b25126ba P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:33.462073 18319 raft_consensus.cc:2258] T 649d43610ba24875a4418e39b25126ba P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:33.469602 18319 ts_tablet_manager.cc:1905] T 649d43610ba24875a4418e39b25126ba P 0397d760589d40d3b92ef141cdfc2dbb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 18:51:33.484164 18319 ts_tablet_manager.cc:1918] T 649d43610ba24875a4418e39b25126ba P 0397d760589d40d3b92ef141cdfc2dbb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260810 18:51:33.484536 18319 log.cc:1196] T 649d43610ba24875a4418e39b25126ba P 0397d760589d40d3b92ef141cdfc2dbb: Deleting WAL directory at /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-1-root/wals/649d43610ba24875a4418e39b25126ba
I20260810 18:51:33.485846 16324 catalog_manager.cc:4841] TS 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): tablet 649d43610ba24875a4418e39b25126ba (table test-workload [id=5aa4952cbee8447ca410249e65af5af6]) successfully deleted
I20260810 18:51:34.522612 16622 consensus_queue.cc:237] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:34.528940 16501 raft_consensus.cc:1261] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:34.528986 16750 raft_consensus.cc:1261] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260810 18:51:34.529508 16573 consensus_peers.cc:487] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb -> Peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387): Couldn't send request to peer 831925dd8b644ebab7eb0d672a94eddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d39f8d52fbd04fc58cb3c20d457bab3d. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:34.530995 18341 consensus_queue.cc:1035] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:51:34.532490 18301 consensus_queue.cc:1035] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:51:34.541971 18301 raft_consensus.cc:2932] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:34.543674 16750 raft_consensus.cc:2932] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:34.545224 16501 raft_consensus.cc:2932] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:34.563753 16343 catalog_manager.cc:5495] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:34.567930 16622 consensus_queue.cc:237] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
W20260810 18:51:34.579443 16573 consensus_peers.cc:487] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb -> Peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387): Couldn't send request to peer 831925dd8b644ebab7eb0d672a94eddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32567b0896594c1186a56d27e36ca0a0. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:34.579684 16750 raft_consensus.cc:1261] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:34.580186 16501 raft_consensus.cc:1261] T 32567b0896594c1186a56d27e36ca0a0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:34.581213 18301 consensus_queue.cc:1035] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:34.583226 18301 consensus_queue.cc:1035] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:34.601946 18301 raft_consensus.cc:2932] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:34.603786 16746 raft_consensus.cc:2932] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:34.608530 16501 raft_consensus.cc:2932] T 32567b0896594c1186a56d27e36ca0a0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:34.620178 16636 consensus_queue.cc:237] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:34.625281 16345 catalog_manager.cc:5495] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
W20260810 18:51:34.634604 16573 consensus_peers.cc:487] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb -> Peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387): Couldn't send request to peer 831925dd8b644ebab7eb0d672a94eddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c87160448924498ac7de68947b3dc19. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:34.635533 16746 raft_consensus.cc:1261] T 4c87160448924498ac7de68947b3dc19 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:34.635644 16506 raft_consensus.cc:1261] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:34.637181 18341 consensus_queue.cc:1035] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:34.639699 18290 consensus_queue.cc:1035] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:51:34.650022 18301 raft_consensus.cc:2932] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:34.650696 16750 raft_consensus.cc:2932] T 4c87160448924498ac7de68947b3dc19 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:34.670130 16501 raft_consensus.cc:2932] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:34.675674 16636 consensus_queue.cc:237] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:34.687359 16344 catalog_manager.cc:5495] T 4c87160448924498ac7de68947b3dc19 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:34.689349 16750 raft_consensus.cc:1261] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:34.690795 18301 consensus_queue.cc:1035] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260810 18:51:34.697069 16573 consensus_peers.cc:487] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb -> Peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387): Couldn't send request to peer 831925dd8b644ebab7eb0d672a94eddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91b6af1458cd44d8ba26b43579613ecb. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:34.705076 18341 raft_consensus.cc:2932] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:34.706303 16501 raft_consensus.cc:1261] T 91b6af1458cd44d8ba26b43579613ecb P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:34.706774 16746 raft_consensus.cc:2932] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:34.709162 18290 consensus_queue.cc:1035] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:34.744169 16501 raft_consensus.cc:2932] T 91b6af1458cd44d8ba26b43579613ecb P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:34.750975 16343 catalog_manager.cc:5495] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:35.075532 18369 ts_tablet_manager.cc:927] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd: Initiating tablet copy from peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:35.077299 18369 tablet_copy_client.cc:316] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Beginning tablet copy session from remote peer at address 127.14.32.194:45827
I20260810 18:51:35.079082 16648 tablet_copy_service.cc:140] P 0397d760589d40d3b92ef141cdfc2dbb: Received BeginTabletCopySession request for tablet 32567b0896594c1186a56d27e36ca0a0 from peer 831925dd8b644ebab7eb0d672a94eddd ({username='slave'} at 127.0.0.1:60066)
I20260810 18:51:35.079610 16648 tablet_copy_service.cc:161] P 0397d760589d40d3b92ef141cdfc2dbb: Beginning new tablet copy session on tablet 32567b0896594c1186a56d27e36ca0a0 from peer 831925dd8b644ebab7eb0d672a94eddd at {username='slave'} at 127.0.0.1:60066: session id = 831925dd8b644ebab7eb0d672a94eddd-32567b0896594c1186a56d27e36ca0a0
I20260810 18:51:35.087697 16648 tablet_copy_source_session.cc:205] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:51:35.089915 18375 ts_tablet_manager.cc:927] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd: Initiating tablet copy from peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:35.091228 18369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32567b0896594c1186a56d27e36ca0a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:35.091848 18375 tablet_copy_client.cc:316] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Beginning tablet copy session from remote peer at address 127.14.32.194:45827
I20260810 18:51:35.093264 16648 tablet_copy_service.cc:140] P 0397d760589d40d3b92ef141cdfc2dbb: Received BeginTabletCopySession request for tablet d39f8d52fbd04fc58cb3c20d457bab3d from peer 831925dd8b644ebab7eb0d672a94eddd ({username='slave'} at 127.0.0.1:60066)
I20260810 18:51:35.093902 16648 tablet_copy_service.cc:161] P 0397d760589d40d3b92ef141cdfc2dbb: Beginning new tablet copy session on tablet d39f8d52fbd04fc58cb3c20d457bab3d from peer 831925dd8b644ebab7eb0d672a94eddd at {username='slave'} at 127.0.0.1:60066: session id = 831925dd8b644ebab7eb0d672a94eddd-d39f8d52fbd04fc58cb3c20d457bab3d
I20260810 18:51:35.101918 16648 tablet_copy_source_session.cc:205] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:51:35.106119 18375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d39f8d52fbd04fc58cb3c20d457bab3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:35.107952 18369 tablet_copy_client.cc:791] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 0 data blocks...
I20260810 18:51:35.108780 18369 tablet_copy_client.cc:655] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 1 WAL segments...
I20260810 18:51:35.114765 18369 tablet_copy_client.cc:531] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:51:35.124961 18369 tablet_bootstrap.cc:492] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap starting.
I20260810 18:51:35.142138 18375 tablet_copy_client.cc:791] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 0 data blocks...
I20260810 18:51:35.143149 18375 tablet_copy_client.cc:655] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 1 WAL segments...
I20260810 18:51:35.147831 18375 tablet_copy_client.cc:531] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:51:35.154826 18369 tablet_bootstrap.cc:492] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:51:35.155823 18369 tablet_bootstrap.cc:492] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap complete.
I20260810 18:51:35.156530 18369 ts_tablet_manager.cc:1397] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.019s
I20260810 18:51:35.157644 18375 tablet_bootstrap.cc:492] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap starting.
I20260810 18:51:35.159835 18369 raft_consensus.cc:348] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:35.160805 18369 raft_consensus.cc:729] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Initialized, Role: LEARNER
I20260810 18:51:35.161617 18369 consensus_queue.cc:260] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:35.172703 18369 ts_tablet_manager.cc:1428] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd: Time spent starting tablet: real 0.016s user 0.006s sys 0.009s
I20260810 18:51:35.187039 16648 tablet_copy_service.cc:342] P 0397d760589d40d3b92ef141cdfc2dbb: Request end of tablet copy session 831925dd8b644ebab7eb0d672a94eddd-32567b0896594c1186a56d27e36ca0a0 received from {username='slave'} at 127.0.0.1:60066
I20260810 18:51:35.187711 16648 tablet_copy_service.cc:433] P 0397d760589d40d3b92ef141cdfc2dbb: ending tablet copy session 831925dd8b644ebab7eb0d672a94eddd-32567b0896594c1186a56d27e36ca0a0 on tablet 32567b0896594c1186a56d27e36ca0a0 with peer 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:35.189976 18375 tablet_bootstrap.cc:492] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:51:35.191534 18375 tablet_bootstrap.cc:492] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap complete.
I20260810 18:51:35.192421 18375 ts_tablet_manager.cc:1397] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd: Time spent bootstrapping tablet: real 0.035s user 0.022s sys 0.011s
I20260810 18:51:35.195724 18375 raft_consensus.cc:348] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:35.196718 18375 raft_consensus.cc:729] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Initialized, Role: LEARNER
I20260810 18:51:35.197459 18375 consensus_queue.cc:260] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:35.201102 18375 ts_tablet_manager.cc:1428] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260810 18:51:35.208722 18369 ts_tablet_manager.cc:927] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd: Initiating tablet copy from peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:35.218673 18369 tablet_copy_client.cc:316] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Beginning tablet copy session from remote peer at address 127.14.32.194:45827
I20260810 18:51:35.219365 16648 tablet_copy_service.cc:342] P 0397d760589d40d3b92ef141cdfc2dbb: Request end of tablet copy session 831925dd8b644ebab7eb0d672a94eddd-d39f8d52fbd04fc58cb3c20d457bab3d received from {username='slave'} at 127.0.0.1:60066
I20260810 18:51:35.219820 16648 tablet_copy_service.cc:433] P 0397d760589d40d3b92ef141cdfc2dbb: ending tablet copy session 831925dd8b644ebab7eb0d672a94eddd-d39f8d52fbd04fc58cb3c20d457bab3d on tablet d39f8d52fbd04fc58cb3c20d457bab3d with peer 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:35.223311 16648 tablet_copy_service.cc:140] P 0397d760589d40d3b92ef141cdfc2dbb: Received BeginTabletCopySession request for tablet 4c87160448924498ac7de68947b3dc19 from peer 831925dd8b644ebab7eb0d672a94eddd ({username='slave'} at 127.0.0.1:60066)
I20260810 18:51:35.223793 16648 tablet_copy_service.cc:161] P 0397d760589d40d3b92ef141cdfc2dbb: Beginning new tablet copy session on tablet 4c87160448924498ac7de68947b3dc19 from peer 831925dd8b644ebab7eb0d672a94eddd at {username='slave'} at 127.0.0.1:60066: session id = 831925dd8b644ebab7eb0d672a94eddd-4c87160448924498ac7de68947b3dc19
I20260810 18:51:35.230672 16648 tablet_copy_source_session.cc:205] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:51:35.233740 18369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c87160448924498ac7de68947b3dc19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:35.237298 18375 ts_tablet_manager.cc:927] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd: Initiating tablet copy from peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:35.239030 18375 tablet_copy_client.cc:316] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Beginning tablet copy session from remote peer at address 127.14.32.194:45827
I20260810 18:51:35.241214 16648 tablet_copy_service.cc:140] P 0397d760589d40d3b92ef141cdfc2dbb: Received BeginTabletCopySession request for tablet 91b6af1458cd44d8ba26b43579613ecb from peer 831925dd8b644ebab7eb0d672a94eddd ({username='slave'} at 127.0.0.1:60066)
I20260810 18:51:35.241701 16648 tablet_copy_service.cc:161] P 0397d760589d40d3b92ef141cdfc2dbb: Beginning new tablet copy session on tablet 91b6af1458cd44d8ba26b43579613ecb from peer 831925dd8b644ebab7eb0d672a94eddd at {username='slave'} at 127.0.0.1:60066: session id = 831925dd8b644ebab7eb0d672a94eddd-91b6af1458cd44d8ba26b43579613ecb
I20260810 18:51:35.247905 16648 tablet_copy_source_session.cc:205] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:51:35.250717 18375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91b6af1458cd44d8ba26b43579613ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:35.255388 18369 tablet_copy_client.cc:791] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 0 data blocks...
I20260810 18:51:35.255981 18369 tablet_copy_client.cc:655] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 1 WAL segments...
I20260810 18:51:35.260699 18369 tablet_copy_client.cc:531] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:51:35.269816 18369 tablet_bootstrap.cc:492] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap starting.
I20260810 18:51:35.274479 18375 tablet_copy_client.cc:791] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 0 data blocks...
I20260810 18:51:35.275164 18375 tablet_copy_client.cc:655] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 1 WAL segments...
I20260810 18:51:35.285319 18375 tablet_copy_client.cc:531] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:51:35.300940 18375 tablet_bootstrap.cc:492] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap starting.
I20260810 18:51:35.308122 18369 tablet_bootstrap.cc:492] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:51:35.309146 18369 tablet_bootstrap.cc:492] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap complete.
I20260810 18:51:35.309893 18369 ts_tablet_manager.cc:1397] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd: Time spent bootstrapping tablet: real 0.040s user 0.028s sys 0.003s
I20260810 18:51:35.313513 18369 raft_consensus.cc:348] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:35.315267 18369 raft_consensus.cc:729] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Initialized, Role: LEARNER
I20260810 18:51:35.315996 18369 consensus_queue.cc:260] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:35.319185 18369 ts_tablet_manager.cc:1428] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd: Time spent starting tablet: real 0.009s user 0.006s sys 0.003s
I20260810 18:51:35.325281 16648 tablet_copy_service.cc:342] P 0397d760589d40d3b92ef141cdfc2dbb: Request end of tablet copy session 831925dd8b644ebab7eb0d672a94eddd-4c87160448924498ac7de68947b3dc19 received from {username='slave'} at 127.0.0.1:60066
I20260810 18:51:35.325707 16648 tablet_copy_service.cc:433] P 0397d760589d40d3b92ef141cdfc2dbb: ending tablet copy session 831925dd8b644ebab7eb0d672a94eddd-4c87160448924498ac7de68947b3dc19 on tablet 4c87160448924498ac7de68947b3dc19 with peer 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:35.330417 18375 tablet_bootstrap.cc:492] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:51:35.331856 18375 tablet_bootstrap.cc:492] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap complete.
I20260810 18:51:35.332510 18375 ts_tablet_manager.cc:1397] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.017s
I20260810 18:51:35.335831 18375 raft_consensus.cc:348] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:35.336743 18375 raft_consensus.cc:729] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Initialized, Role: LEARNER
I20260810 18:51:35.337471 18375 consensus_queue.cc:260] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:35.340027 18375 ts_tablet_manager.cc:1428] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 18:51:35.346820 16648 tablet_copy_service.cc:342] P 0397d760589d40d3b92ef141cdfc2dbb: Request end of tablet copy session 831925dd8b644ebab7eb0d672a94eddd-91b6af1458cd44d8ba26b43579613ecb received from {username='slave'} at 127.0.0.1:60066
I20260810 18:51:35.347245 16648 tablet_copy_service.cc:433] P 0397d760589d40d3b92ef141cdfc2dbb: ending tablet copy session 831925dd8b644ebab7eb0d672a94eddd-91b6af1458cd44d8ba26b43579613ecb on tablet 91b6af1458cd44d8ba26b43579613ecb with peer 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:35.457290 17785 raft_consensus.cc:1203] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260810 18:51:35.595845 17785 raft_consensus.cc:1203] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260810 18:51:35.611761 17785 raft_consensus.cc:1203] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260810 18:51:35.664978 17785 raft_consensus.cc:1203] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260810 18:51:35.982146 18341 raft_consensus.cc:1050] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb: attempting to promote NON_VOTER 831925dd8b644ebab7eb0d672a94eddd to VOTER
I20260810 18:51:35.984813 18341 consensus_queue.cc:237] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:35.992178 16506 raft_consensus.cc:1261] T 32567b0896594c1186a56d27e36ca0a0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:35.993443 16746 raft_consensus.cc:1261] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:35.993391 18341 consensus_queue.cc:1035] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:35.994889 18301 consensus_queue.cc:1035] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:35.995268 17785 raft_consensus.cc:1261] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:35.997085 18290 consensus_queue.cc:1035] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:36.018862 18341 raft_consensus.cc:2932] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.021597 16506 raft_consensus.cc:2932] T 32567b0896594c1186a56d27e36ca0a0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.023391 17785 raft_consensus.cc:2932] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.039696 16343 catalog_manager.cc:5495] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:51:36.043870 16746 raft_consensus.cc:2932] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.057804 18301 raft_consensus.cc:1050] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb: attempting to promote NON_VOTER 831925dd8b644ebab7eb0d672a94eddd to VOTER
I20260810 18:51:36.094610 18301 consensus_queue.cc:237] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:36.108537 18402 raft_consensus.cc:1050] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb: attempting to promote NON_VOTER 831925dd8b644ebab7eb0d672a94eddd to VOTER
I20260810 18:51:36.112183 18402 consensus_queue.cc:237] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:36.117130 16622 consensus_queue.cc:237] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:36.138626 17785 raft_consensus.cc:1261] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:36.142138 18403 consensus_queue.cc:1035] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:51:36.142479 18341 raft_consensus.cc:1050] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb: attempting to promote NON_VOTER 831925dd8b644ebab7eb0d672a94eddd to VOTER
I20260810 18:51:36.144953 18341 consensus_queue.cc:237] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:36.168972 16743 raft_consensus.cc:1261] T 4c87160448924498ac7de68947b3dc19 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:36.227586 17786 raft_consensus.cc:1261] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:36.169299 16750 raft_consensus.cc:1261] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:36.219533 17783 raft_consensus.cc:1261] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:36.170060 16745 raft_consensus.cc:1261] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:36.229713 16506 raft_consensus.cc:1261] T 91b6af1458cd44d8ba26b43579613ecb P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:36.234071 16506 raft_consensus.cc:1261] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:36.236796 17783 raft_consensus.cc:1261] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:36.237660 18414 consensus_queue.cc:1035] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:36.170245 16749 raft_consensus.cc:1261] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:36.238078 18403 consensus_queue.cc:1035] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:36.238858 18411 consensus_queue.cc:1035] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:36.241667 16506 raft_consensus.cc:1261] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:36.241747 18341 consensus_queue.cc:1035] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:36.243690 18411 consensus_queue.cc:1035] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:36.244846 18341 consensus_queue.cc:1035] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:36.245432 18290 consensus_queue.cc:1035] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:36.245527 18403 consensus_queue.cc:1035] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:36.250502 18414 consensus_queue.cc:1035] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:51:36.253132 18403 consensus_queue.cc:1035] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:36.280416 18341 raft_consensus.cc:2932] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.305953 16324 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 32567b0896594c1186a56d27e36ca0a0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 18:51:36.307878 17787 raft_consensus.cc:2932] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.310118 18341 raft_consensus.cc:2932] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.312868 16345 catalog_manager.cc:5495] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:36.326500 18290 raft_consensus.cc:2932] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.329893 17786 raft_consensus.cc:2932] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
W20260810 18:51:36.339982 18432 log.cc:924] Time spent T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20260810 18:51:36.340736 16492 tablet_service.cc:1514] Processing DeleteTablet for tablet 32567b0896594c1186a56d27e36ca0a0 with delete_type TABLET_DATA_TOMBSTONED (TS d304ac47fdf846de81337057a4ef7bea not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52330
I20260810 18:51:36.348728 17785 raft_consensus.cc:2932] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.355057 16344 catalog_manager.cc:5495] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:51:36.358163 16750 raft_consensus.cc:2932] T 4c87160448924498ac7de68947b3dc19 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.359086 18411 raft_consensus.cc:2932] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.370362 16506 raft_consensus.cc:2932] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.370635 16749 raft_consensus.cc:2932] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.372159 16746 raft_consensus.cc:2932] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.383749 16745 raft_consensus.cc:2932] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.392841 16631 consensus_queue.cc:237] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:36.395695 18439 tablet_replica.cc:330] T 32567b0896594c1186a56d27e36ca0a0 P d304ac47fdf846de81337057a4ef7bea: stopping tablet replica
I20260810 18:51:36.407476 16344 catalog_manager.cc:5495] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:51:36.410740 18439 raft_consensus.cc:2229] T 32567b0896594c1186a56d27e36ca0a0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:36.411517 18439 raft_consensus.cc:2258] T 32567b0896594c1186a56d27e36ca0a0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:36.419386 17787 raft_consensus.cc:1261] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:36.426240 17785 raft_consensus.cc:2932] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.434857 16505 raft_consensus.cc:1261] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:36.441118 18403 consensus_queue.cc:1035] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:51:36.446862 18403 consensus_queue.cc:1035] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:51:36.449775 16506 raft_consensus.cc:2932] T 91b6af1458cd44d8ba26b43579613ecb P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.453861 18341 raft_consensus.cc:2932] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.463536 17785 raft_consensus.cc:2932] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.465049 18439 ts_tablet_manager.cc:1905] T 32567b0896594c1186a56d27e36ca0a0 P d304ac47fdf846de81337057a4ef7bea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 18:51:36.467343 16501 raft_consensus.cc:2932] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.483723 16324 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 4c87160448924498ac7de68947b3dc19 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 18:51:36.486069 16343 catalog_manager.cc:5495] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.512333 18439 ts_tablet_manager.cc:1918] T 32567b0896594c1186a56d27e36ca0a0 P d304ac47fdf846de81337057a4ef7bea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260810 18:51:36.489614 16506 raft_consensus.cc:2932] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.512838 18439 log.cc:1196] T 32567b0896594c1186a56d27e36ca0a0 P d304ac47fdf846de81337057a4ef7bea: Deleting WAL directory at /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-0-root/wals/32567b0896594c1186a56d27e36ca0a0
I20260810 18:51:36.515617 16323 catalog_manager.cc:4841] TS d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): tablet 32567b0896594c1186a56d27e36ca0a0 (table test-workload [id=5aa4952cbee8447ca410249e65af5af6]) successfully deleted
I20260810 18:51:36.516474 16343 catalog_manager.cc:5495] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: config changed from index 3 to 4, VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) evicted. New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.537876 16726 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c87160448924498ac7de68947b3dc19 with delete_type TABLET_DATA_TOMBSTONED (TS 980f343dd1fd4e049143f8b9d9f2f1be not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43720
I20260810 18:51:36.546648 18443 tablet_replica.cc:330] T 4c87160448924498ac7de68947b3dc19 P 980f343dd1fd4e049143f8b9d9f2f1be: stopping tablet replica
I20260810 18:51:36.550711 18443 raft_consensus.cc:2229] T 4c87160448924498ac7de68947b3dc19 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:36.551534 18443 raft_consensus.cc:2258] T 4c87160448924498ac7de68947b3dc19 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:36.552488 16622 consensus_queue.cc:237] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:36.552794 16631 consensus_queue.cc:237] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:36.555315 18443 ts_tablet_manager.cc:1905] T 4c87160448924498ac7de68947b3dc19 P 980f343dd1fd4e049143f8b9d9f2f1be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 18:51:36.558761 17785 raft_consensus.cc:1261] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:36.558964 16746 raft_consensus.cc:1261] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:36.560871 17785 raft_consensus.cc:1261] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:36.561020 18341 consensus_queue.cc:1035] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:51:36.562208 16506 raft_consensus.cc:1261] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:36.562455 18403 consensus_queue.cc:1035] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:51:36.566259 18403 consensus_queue.cc:1035] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:51:36.567620 18341 consensus_queue.cc:1035] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:51:36.569844 18301 raft_consensus.cc:2932] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.571394 17785 raft_consensus.cc:2932] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.575839 16746 raft_consensus.cc:2932] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.582105 18414 raft_consensus.cc:2932] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.584931 16344 catalog_manager.cc:5495] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.589922 17786 raft_consensus.cc:2932] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.596730 18443 ts_tablet_manager.cc:1918] T 4c87160448924498ac7de68947b3dc19 P 980f343dd1fd4e049143f8b9d9f2f1be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260810 18:51:36.596829 16492 tablet_service.cc:1514] Processing DeleteTablet for tablet 91b6af1458cd44d8ba26b43579613ecb with delete_type TABLET_DATA_TOMBSTONED (TS d304ac47fdf846de81337057a4ef7bea not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52330
I20260810 18:51:36.597332 18443 log.cc:1196] T 4c87160448924498ac7de68947b3dc19 P 980f343dd1fd4e049143f8b9d9f2f1be: Deleting WAL directory at /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-2-root/wals/4c87160448924498ac7de68947b3dc19
I20260810 18:51:36.597913 18439 tablet_replica.cc:330] T 91b6af1458cd44d8ba26b43579613ecb P d304ac47fdf846de81337057a4ef7bea: stopping tablet replica
I20260810 18:51:36.599004 18439 raft_consensus.cc:2229] T 91b6af1458cd44d8ba26b43579613ecb P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:36.599713 18439 raft_consensus.cc:2258] T 91b6af1458cd44d8ba26b43579613ecb P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:36.600976 16328 catalog_manager.cc:4841] TS 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): tablet 4c87160448924498ac7de68947b3dc19 (table test-workload [id=5aa4952cbee8447ca410249e65af5af6]) successfully deleted
I20260810 18:51:36.601311 16324 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 91b6af1458cd44d8ba26b43579613ecb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 18:51:36.603215 16324 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d39f8d52fbd04fc58cb3c20d457bab3d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 18:51:36.605278 18439 ts_tablet_manager.cc:1905] T 91b6af1458cd44d8ba26b43579613ecb P d304ac47fdf846de81337057a4ef7bea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 18:51:36.616335 16506 raft_consensus.cc:2932] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.620510 16343 catalog_manager.cc:5495] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: config changed from index 3 to 4, VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) evicted. New cstate: current_term: 1 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:36.625681 18439 ts_tablet_manager.cc:1918] T 91b6af1458cd44d8ba26b43579613ecb P d304ac47fdf846de81337057a4ef7bea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260810 18:51:36.626148 18439 log.cc:1196] T 91b6af1458cd44d8ba26b43579613ecb P d304ac47fdf846de81337057a4ef7bea: Deleting WAL directory at /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-0-root/wals/91b6af1458cd44d8ba26b43579613ecb
I20260810 18:51:36.637639 16323 catalog_manager.cc:4841] TS d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): tablet 91b6af1458cd44d8ba26b43579613ecb (table test-workload [id=5aa4952cbee8447ca410249e65af5af6]) successfully deleted
I20260810 18:51:36.638417 16726 tablet_service.cc:1514] Processing DeleteTablet for tablet d39f8d52fbd04fc58cb3c20d457bab3d with delete_type TABLET_DATA_TOMBSTONED (TS 980f343dd1fd4e049143f8b9d9f2f1be not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43720
I20260810 18:51:36.639285 18443 tablet_replica.cc:330] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be: stopping tablet replica
I20260810 18:51:36.640236 18443 raft_consensus.cc:2229] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:36.640961 18443 raft_consensus.cc:2258] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:36.644304 18443 ts_tablet_manager.cc:1905] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 18:51:36.658310 18443 ts_tablet_manager.cc:1918] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260810 18:51:36.658751 18443 log.cc:1196] T d39f8d52fbd04fc58cb3c20d457bab3d P 980f343dd1fd4e049143f8b9d9f2f1be: Deleting WAL directory at /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-2-root/wals/d39f8d52fbd04fc58cb3c20d457bab3d
I20260810 18:51:36.660144 16328 catalog_manager.cc:4841] TS 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): tablet d39f8d52fbd04fc58cb3c20d457bab3d (table test-workload [id=5aa4952cbee8447ca410249e65af5af6]) successfully deleted
I20260810 18:51:37.716648 16506 consensus_queue.cc:237] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:37.724227 16746 raft_consensus.cc:1261] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:37.724231 16622 raft_consensus.cc:1261] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:37.726701 18381 consensus_queue.cc:1035] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:51:37.727643 18419 consensus_queue.cc:1035] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260810 18:51:37.729864 16447 consensus_peers.cc:487] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea -> Peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387): Couldn't send request to peer 831925dd8b644ebab7eb0d672a94eddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cdfc652dd163415b94d3f0bc3931a07f. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:37.747308 18419 raft_consensus.cc:2932] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:37.748862 16746 raft_consensus.cc:2932] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:37.761453 16746 consensus_queue.cc:237] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:37.766256 16622 raft_consensus.cc:2932] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:37.767256 16344 catalog_manager.cc:5495] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:37.781090 16631 raft_consensus.cc:1261] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:37.782680 18445 consensus_queue.cc:1035] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:51:37.784701 16506 raft_consensus.cc:1261] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:37.786229 18445 consensus_queue.cc:1035] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260810 18:51:37.788182 16695 consensus_peers.cc:487] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be -> Peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387): Couldn't send request to peer 831925dd8b644ebab7eb0d672a94eddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ef7c65533724e678b75830002906e2c. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:37.811466 18436 raft_consensus.cc:2932] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:37.815397 16622 raft_consensus.cc:2932] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:37.820135 16501 raft_consensus.cc:2932] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:37.835670 16342 catalog_manager.cc:5495] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:37.837993 16749 consensus_queue.cc:237] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
W20260810 18:51:37.847807 16695 consensus_peers.cc:487] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be -> Peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387): Couldn't send request to peer 831925dd8b644ebab7eb0d672a94eddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03cb267b222b440981c94d431e7273bf. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:37.848441 16622 raft_consensus.cc:1261] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:37.849182 16501 raft_consensus.cc:1261] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:37.850508 18442 consensus_queue.cc:1035] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:37.851527 18436 consensus_queue.cc:1035] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:37.860596 18442 raft_consensus.cc:2932] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:37.862115 16622 raft_consensus.cc:2932] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:37.862566 16501 raft_consensus.cc:2932] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:37.880349 16501 consensus_queue.cc:237] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:37.880625 16342 catalog_manager.cc:5495] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:37.890030 16749 raft_consensus.cc:1261] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:37.891860 16622 raft_consensus.cc:1261] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:51:37.893780 18446 consensus_queue.cc:1035] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:37.894835 18381 consensus_queue.cc:1035] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260810 18:51:37.907935 16447 consensus_peers.cc:487] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea -> Peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387): Couldn't send request to peer 831925dd8b644ebab7eb0d672a94eddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8810b6b0b676405daf5ffddc50a4d21f. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:51:37.910687 18381 raft_consensus.cc:2932] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:37.915001 16623 raft_consensus.cc:2932] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:37.916324 16746 raft_consensus.cc:2932] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } }
I20260810 18:51:37.928884 16345 catalog_manager.cc:5495] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea reported cstate change: config changed from index -1 to 2, NON_VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) added. New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:38.260885 18474 ts_tablet_manager.cc:927] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd: Initiating tablet copy from peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:38.267796 18474 tablet_copy_client.cc:316] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Beginning tablet copy session from remote peer at address 127.14.32.195:40171
I20260810 18:51:38.269778 16764 tablet_copy_service.cc:140] P 980f343dd1fd4e049143f8b9d9f2f1be: Received BeginTabletCopySession request for tablet 03cb267b222b440981c94d431e7273bf from peer 831925dd8b644ebab7eb0d672a94eddd ({username='slave'} at 127.0.0.1:37716)
I20260810 18:51:38.270310 16764 tablet_copy_service.cc:161] P 980f343dd1fd4e049143f8b9d9f2f1be: Beginning new tablet copy session on tablet 03cb267b222b440981c94d431e7273bf from peer 831925dd8b644ebab7eb0d672a94eddd at {username='slave'} at 127.0.0.1:37716: session id = 831925dd8b644ebab7eb0d672a94eddd-03cb267b222b440981c94d431e7273bf
I20260810 18:51:38.278995 16764 tablet_copy_source_session.cc:205] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:51:38.282698 18474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03cb267b222b440981c94d431e7273bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:38.299237 18474 tablet_copy_client.cc:791] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 0 data blocks...
I20260810 18:51:38.299860 18474 tablet_copy_client.cc:655] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 1 WAL segments...
I20260810 18:51:38.307749 18474 tablet_copy_client.cc:531] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:51:38.316478 18474 tablet_bootstrap.cc:492] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap starting.
I20260810 18:51:38.317848 18478 ts_tablet_manager.cc:927] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd: Initiating tablet copy from peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791)
I20260810 18:51:38.320269 18478 tablet_copy_client.cc:316] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Beginning tablet copy session from remote peer at address 127.14.32.193:40791
I20260810 18:51:38.323498 16525 tablet_copy_service.cc:140] P d304ac47fdf846de81337057a4ef7bea: Received BeginTabletCopySession request for tablet cdfc652dd163415b94d3f0bc3931a07f from peer 831925dd8b644ebab7eb0d672a94eddd ({username='slave'} at 127.0.0.1:56468)
I20260810 18:51:38.324091 16525 tablet_copy_service.cc:161] P d304ac47fdf846de81337057a4ef7bea: Beginning new tablet copy session on tablet cdfc652dd163415b94d3f0bc3931a07f from peer 831925dd8b644ebab7eb0d672a94eddd at {username='slave'} at 127.0.0.1:56468: session id = 831925dd8b644ebab7eb0d672a94eddd-cdfc652dd163415b94d3f0bc3931a07f
I20260810 18:51:38.332489 16525 tablet_copy_source_session.cc:205] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:51:38.336520 18478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdfc652dd163415b94d3f0bc3931a07f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:38.340584 18474 tablet_bootstrap.cc:492] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:51:38.341514 18474 tablet_bootstrap.cc:492] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap complete.
I20260810 18:51:38.342195 18474 ts_tablet_manager.cc:1397] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.013s
I20260810 18:51:38.344971 18474 raft_consensus.cc:348] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:38.345924 18474 raft_consensus.cc:729] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Initialized, Role: LEARNER
I20260810 18:51:38.346587 18474 consensus_queue.cc:260] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:38.349875 18474 ts_tablet_manager.cc:1428] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd: Time spent starting tablet: real 0.007s user 0.008s sys 0.001s
I20260810 18:51:38.354329 16764 tablet_copy_service.cc:342] P 980f343dd1fd4e049143f8b9d9f2f1be: Request end of tablet copy session 831925dd8b644ebab7eb0d672a94eddd-03cb267b222b440981c94d431e7273bf received from {username='slave'} at 127.0.0.1:37716
I20260810 18:51:38.354704 16764 tablet_copy_service.cc:433] P 980f343dd1fd4e049143f8b9d9f2f1be: ending tablet copy session 831925dd8b644ebab7eb0d672a94eddd-03cb267b222b440981c94d431e7273bf on tablet 03cb267b222b440981c94d431e7273bf with peer 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:38.355675 18478 tablet_copy_client.cc:791] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 0 data blocks...
I20260810 18:51:38.356698 18478 tablet_copy_client.cc:655] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 1 WAL segments...
I20260810 18:51:38.360952 18478 tablet_copy_client.cc:531] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:51:38.368459 18478 tablet_bootstrap.cc:492] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap starting.
I20260810 18:51:38.385468 18474 ts_tablet_manager.cc:927] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd: Initiating tablet copy from peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:38.388363 18474 tablet_copy_client.cc:316] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Beginning tablet copy session from remote peer at address 127.14.32.195:40171
I20260810 18:51:38.389077 18478 tablet_bootstrap.cc:492] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:51:38.389787 18478 tablet_bootstrap.cc:492] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap complete.
I20260810 18:51:38.389907 16764 tablet_copy_service.cc:140] P 980f343dd1fd4e049143f8b9d9f2f1be: Received BeginTabletCopySession request for tablet 7ef7c65533724e678b75830002906e2c from peer 831925dd8b644ebab7eb0d672a94eddd ({username='slave'} at 127.0.0.1:37716)
I20260810 18:51:38.390393 18478 ts_tablet_manager.cc:1397] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.003s
I20260810 18:51:38.390470 16764 tablet_copy_service.cc:161] P 980f343dd1fd4e049143f8b9d9f2f1be: Beginning new tablet copy session on tablet 7ef7c65533724e678b75830002906e2c from peer 831925dd8b644ebab7eb0d672a94eddd at {username='slave'} at 127.0.0.1:37716: session id = 831925dd8b644ebab7eb0d672a94eddd-7ef7c65533724e678b75830002906e2c
I20260810 18:51:38.392964 18478 raft_consensus.cc:348] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:38.393888 18478 raft_consensus.cc:729] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Initialized, Role: LEARNER
I20260810 18:51:38.394611 18478 consensus_queue.cc:260] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:38.397015 18478 ts_tablet_manager.cc:1428] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260810 18:51:38.400620 16525 tablet_copy_service.cc:342] P d304ac47fdf846de81337057a4ef7bea: Request end of tablet copy session 831925dd8b644ebab7eb0d672a94eddd-cdfc652dd163415b94d3f0bc3931a07f received from {username='slave'} at 127.0.0.1:56468
I20260810 18:51:38.400965 16525 tablet_copy_service.cc:433] P d304ac47fdf846de81337057a4ef7bea: ending tablet copy session 831925dd8b644ebab7eb0d672a94eddd-cdfc652dd163415b94d3f0bc3931a07f on tablet cdfc652dd163415b94d3f0bc3931a07f with peer 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:38.406940 16764 tablet_copy_source_session.cc:205] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:51:38.410856 18474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ef7c65533724e678b75830002906e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:38.432268 18474 tablet_copy_client.cc:791] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 0 data blocks...
I20260810 18:51:38.432932 18474 tablet_copy_client.cc:655] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 1 WAL segments...
I20260810 18:51:38.436455 18474 tablet_copy_client.cc:531] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:51:38.445323 18474 tablet_bootstrap.cc:492] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap starting.
I20260810 18:51:38.466264 18474 tablet_bootstrap.cc:492] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:51:38.467284 18474 tablet_bootstrap.cc:492] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap complete.
I20260810 18:51:38.467989 18474 ts_tablet_manager.cc:1397] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd: Time spent bootstrapping tablet: real 0.023s user 0.022s sys 0.000s
I20260810 18:51:38.470804 18474 raft_consensus.cc:348] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:38.471983 18474 raft_consensus.cc:729] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Initialized, Role: LEARNER
I20260810 18:51:38.472903 18474 consensus_queue.cc:260] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:38.481451 18474 ts_tablet_manager.cc:1428] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260810 18:51:38.483345 16764 tablet_copy_service.cc:342] P 980f343dd1fd4e049143f8b9d9f2f1be: Request end of tablet copy session 831925dd8b644ebab7eb0d672a94eddd-7ef7c65533724e678b75830002906e2c received from {username='slave'} at 127.0.0.1:37716
I20260810 18:51:38.483744 16764 tablet_copy_service.cc:433] P 980f343dd1fd4e049143f8b9d9f2f1be: ending tablet copy session 831925dd8b644ebab7eb0d672a94eddd-7ef7c65533724e678b75830002906e2c on tablet 7ef7c65533724e678b75830002906e2c with peer 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:38.488220 18478 ts_tablet_manager.cc:927] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd: Initiating tablet copy from peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791)
I20260810 18:51:38.491106 18478 tablet_copy_client.cc:316] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Beginning tablet copy session from remote peer at address 127.14.32.193:40791
I20260810 18:51:38.492663 16525 tablet_copy_service.cc:140] P d304ac47fdf846de81337057a4ef7bea: Received BeginTabletCopySession request for tablet 8810b6b0b676405daf5ffddc50a4d21f from peer 831925dd8b644ebab7eb0d672a94eddd ({username='slave'} at 127.0.0.1:56468)
I20260810 18:51:38.493162 16525 tablet_copy_service.cc:161] P d304ac47fdf846de81337057a4ef7bea: Beginning new tablet copy session on tablet 8810b6b0b676405daf5ffddc50a4d21f from peer 831925dd8b644ebab7eb0d672a94eddd at {username='slave'} at 127.0.0.1:56468: session id = 831925dd8b644ebab7eb0d672a94eddd-8810b6b0b676405daf5ffddc50a4d21f
I20260810 18:51:38.499748 16525 tablet_copy_source_session.cc:205] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:51:38.505105 18478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8810b6b0b676405daf5ffddc50a4d21f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:51:38.522097 18478 tablet_copy_client.cc:791] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 0 data blocks...
I20260810 18:51:38.524477 18478 tablet_copy_client.cc:655] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Starting download of 1 WAL segments...
I20260810 18:51:38.546563 18478 tablet_copy_client.cc:531] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:51:38.554567 18478 tablet_bootstrap.cc:492] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap starting.
I20260810 18:51:38.573529 18478 tablet_bootstrap.cc:492] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:51:38.574393 18478 tablet_bootstrap.cc:492] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd: Bootstrap complete.
I20260810 18:51:38.574980 18478 ts_tablet_manager.cc:1397] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.008s
I20260810 18:51:38.577443 18478 raft_consensus.cc:348] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:38.578238 18478 raft_consensus.cc:729] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Initialized, Role: LEARNER
I20260810 18:51:38.578780 18478 consensus_queue.cc:260] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:38.581344 18478 ts_tablet_manager.cc:1428] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:51:38.582937 16525 tablet_copy_service.cc:342] P d304ac47fdf846de81337057a4ef7bea: Request end of tablet copy session 831925dd8b644ebab7eb0d672a94eddd-8810b6b0b676405daf5ffddc50a4d21f received from {username='slave'} at 127.0.0.1:56468
I20260810 18:51:38.583317 16525 tablet_copy_service.cc:433] P d304ac47fdf846de81337057a4ef7bea: ending tablet copy session 831925dd8b644ebab7eb0d672a94eddd-8810b6b0b676405daf5ffddc50a4d21f on tablet 8810b6b0b676405daf5ffddc50a4d21f with peer 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:38.683761 17783 raft_consensus.cc:1203] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260810 18:51:38.751227 17783 raft_consensus.cc:1203] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260810 18:51:38.963757 17783 raft_consensus.cc:1203] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260810 18:51:38.975631 17786 raft_consensus.cc:1203] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260810 18:51:39.947166 18507 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:39.947857 18507 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:39.963186 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:39.975286 18507 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:39.983569 18503 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:39.984251 18503 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:39.992061 18503 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:39.992704 16344 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55260 (request call id 115) took 1011 ms. Trace:
I20260810 18:51:39.993088 16344 rpcz_store.cc:276] 0810 18:51:38.981128 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:38.981258 (+ 130us) service_pool.cc:227] Handling call
0810 18:51:39.992593 (+1011335us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1,"spinlock_wait_cycles":17536}
I20260810 18:51:40.030782 18497 raft_consensus.cc:482] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:40.033169 18497 raft_consensus.cc:504] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.044691 18497 leader_election.cc:288] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:40.057905 18522 raft_consensus.cc:482] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:40.058156 18520 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:40.060271 18520 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.070516 18520 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:40.070955 18505 raft_consensus.cc:482] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:40.071412 18500 raft_consensus.cc:482] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:40.072467 18505 raft_consensus.cc:504] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.072851 18500 raft_consensus.cc:504] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.064057 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:40.076103 18500 leader_election.cc:288] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:40.060465 18522 raft_consensus.cc:504] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.086794 18502 raft_consensus.cc:482] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:40.095585 18502 raft_consensus.cc:504] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.078526 18505 leader_election.cc:288] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:40.110734 18508 raft_consensus.cc:482] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:40.120651 18508 raft_consensus.cc:504] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.121651 18494 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:40.122514 18494 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:40.125272 18523 raft_consensus.cc:482] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:40.125726 18502 leader_election.cc:288] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:40.126140 18523 raft_consensus.cc:504] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.126983 18522 leader_election.cc:288] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:40.129899 18504 raft_consensus.cc:482] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:40.130602 18504 raft_consensus.cc:504] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.112777 18513 raft_consensus.cc:482] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:40.127995 18505 raft_consensus.cc:482] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:40.147719 18507 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:40.151639 18503 raft_consensus.cc:1050] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be: attempting to promote NON_VOTER 831925dd8b644ebab7eb0d672a94eddd to VOTER
I20260810 18:51:40.136197 18513 raft_consensus.cc:504] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.173056 18526 raft_consensus.cc:482] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:40.173276 18505 raft_consensus.cc:504] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:40.174331 18495 raft_consensus.cc:1050] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be: attempting to promote NON_VOTER 831925dd8b644ebab7eb0d672a94eddd to VOTER
I20260810 18:51:40.173800 18526 raft_consensus.cc:504] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.184922 18504 leader_election.cc:288] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:40.184792 18523 leader_election.cc:288] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:40.145057 18522 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:40.193596 18499 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:40.189719 18526 leader_election.cc:288] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:40.195156 18522 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.198592 18522 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:40.200053 18526 raft_consensus.cc:482] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:40.200466 18508 leader_election.cc:288] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:40.203224 18513 leader_election.cc:288] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:40.204244 18502 raft_consensus.cc:482] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:40.171177 18507 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:40.210608 18494 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:40.193475 18495 consensus_queue.cc:237] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:40.182973 18505 leader_election.cc:288] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:40.228268 18531 raft_consensus.cc:482] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:40.228926 18531 raft_consensus.cc:504] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.232357 18531 leader_election.cc:288] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:40.223565 18520 raft_consensus.cc:482] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:40.223385 18502 raft_consensus.cc:504] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.233858 18520 raft_consensus.cc:504] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.236276 18502 leader_election.cc:288] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:40.237462 18520 leader_election.cc:288] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:40.245586 18524 raft_consensus.cc:482] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:40.247057 18507 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:40.222179 18526 raft_consensus.cc:504] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.247831 18507 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:40.248576 18507 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:40.250250 18526 leader_election.cc:288] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:40.220482 18522 raft_consensus.cc:482] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:40.263017 18507 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:40.262699 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:40.264074 18507 raft_consensus.cc:482] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:40.263378 18494 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:40.264058 18497 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:40.264950 18494 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.264853 18507 raft_consensus.cc:504] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:40.265126 18497 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:40.268575 18494 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:40.269301 18500 raft_consensus.cc:482] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:40.270051 18507 leader_election.cc:288] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:40.194380 18499 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.211400 18503 consensus_queue.cc:237] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:40.260807 18505 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:40.246421 18524 raft_consensus.cc:504] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.303447 18507 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:40.350857 18531 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:40.350837 18537 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:40.270152 18500 raft_consensus.cc:504] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.356477 18529 raft_consensus.cc:482] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:40.276088 18522 raft_consensus.cc:504] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.361948 18507 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:40.377058 18528 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:40.377731 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:40.347241 18513 raft_consensus.cc:482] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:40.358083 18499 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:40.379839 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:40.377831 18528 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.383240 18507 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:40.365381 18494 raft_consensus.cc:482] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:40.362493 18505 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:40.363842 18531 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
W20260810 18:51:40.385306 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 88822.7us
I20260810 18:51:40.508742 18512 raft_consensus.cc:482] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:51:40.360540 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 102533us
I20260810 18:51:40.557653 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:40.517570 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:40.385233 18538 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:40.560786 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:40.561717 18522 leader_election.cc:288] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:40.385182 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:40.560784 16624 raft_consensus.cc:2454] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:40.561529 18546 raft_consensus.cc:482] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:40.523128 18516 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:40.577679 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:40.524139 18524 leader_election.cc:288] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:40.378679 18526 raft_consensus.cc:482] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:40.453310 18520 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:40.586710 16636 raft_consensus.cc:2454] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:40.542793 18497 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:40.385887 18535 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:40.452829 18523 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:40.522816 18536 raft_consensus.cc:482] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:40.567420 18499 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:40.453353 18540 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:40.383996 18537 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:40.479460 18534 raft_consensus.cc:482] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:40.386035 18513 raft_consensus.cc:504] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.575601 18546 raft_consensus.cc:504] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.590906 18543 raft_consensus.cc:482] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:40.567582 18507 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:40.561622 18538 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:40.588373 18524 raft_consensus.cc:482] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:40.567230 18494 raft_consensus.cc:504] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:40.606551 18499 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.616847 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:40.617457 18507 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:40.593034 18523 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:40.604519 18535 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:40.608614 18536 raft_consensus.cc:504] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.606731 18520 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:40.605621 18522 raft_consensus.cc:482] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:40.649534 18539 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:40.594245 18504 raft_consensus.cc:482] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:40.592864 18500 leader_election.cc:288] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:40.593227 18497 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:40.621207 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:40.588721 18526 raft_consensus.cc:504] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.556572 18512 raft_consensus.cc:504] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.621721 18534 raft_consensus.cc:504] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.383450 18529 raft_consensus.cc:504] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:51:40.594599 18516 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.736573 18513 leader_election.cc:288] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:40.925942 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:40.777949 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:40.928278 16506 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:40.752168 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:40.928182 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:40.848389 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:40.736703 18494 leader_election.cc:288] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:40.941951 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:40.978543 16750 raft_consensus.cc:2454] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 1.
I20260810 18:51:40.980337 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260810 18:51:40.980844 16783 debug-util.cc:398] Leaking SignalData structure 0x7b08000bdbe0 after lost signal to thread 14470
I20260810 18:51:40.981027 18538 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:40.987499 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:40.992950 16636 raft_consensus.cc:2454] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:40.977687 18500 raft_consensus.cc:482] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:40.994025 18500 raft_consensus.cc:504] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.615164 18540 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:40.927520 18505 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:40.900220 18537 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:51:40.990710 16783 debug-util.cc:398] Leaking SignalData structure 0x7b08001fc2e0 after lost signal to thread 16391
I20260810 18:51:40.836810 18508 raft_consensus.cc:482] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:40.978135 18531 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:40.961398 18513 raft_consensus.cc:482] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:40.871917 18552 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:40.747211 18524 raft_consensus.cc:504] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.972353 18504 raft_consensus.cc:504] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.756208 18554 raft_consensus.cc:482] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:40.870088 18551 raft_consensus.cc:482] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:40.847821 18557 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:40.971972 18522 raft_consensus.cc:504] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:41.007259 18505 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:41.024919 18526 leader_election.cc:288] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:41.009545 18510 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:41.012609 18531 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:41.011641 18513 raft_consensus.cc:504] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:40.983210 18497 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
W20260810 18:51:41.037367 16447 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.021s sys 0.016s
I20260810 18:51:41.008641 18537 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:41.012166 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:41.047361 16447 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.588s user 0.022s sys 0.016s
I20260810 18:51:41.015947 18528 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:40.994813 18539 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:41.011813 18508 raft_consensus.cc:504] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:41.046167 18537 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:40.763674 18543 raft_consensus.cc:504] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:41.047479 16783 debug-util.cc:398] Leaking SignalData structure 0x7b080025e120 after lost signal to thread 16545
I20260810 18:51:41.064079 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.031301 18552 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:41.050108 18551 raft_consensus.cc:504] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:41.059110 16624 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
W20260810 18:51:41.071907 16783 debug-util.cc:398] Leaking SignalData structure 0x7b0800305760 after lost signal to thread 16671
I20260810 18:51:41.032819 17726 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:41.072304 18536 leader_election.cc:288] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:41.068917 18500 leader_election.cc:288] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:41.056792 18546 leader_election.cc:288] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:41.043726 18505 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:41.048590 18531 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:41.096714 18516 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:41.047348 18557 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:41.062197 18534 leader_election.cc:288] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:41.063508 18535 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:41.102447 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.103295 16501 raft_consensus.cc:2454] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:41.082094 18507 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:51:41.082695 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 63921us
I20260810 18:51:41.094767 18529 leader_election.cc:288] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:41.116138 18540 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:41.076943 18560 raft_consensus.cc:482] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:41.103044 18508 leader_election.cc:288] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:41.122812 18543 leader_election.cc:288] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:41.113618 18507 raft_consensus.cc:2726] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 18:51:41.123736 16572 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.014s sys 0.014s
W20260810 18:51:41.124584 16448 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.760s user 0.007s sys 0.025s
I20260810 18:51:41.109645 18523 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:51:41.133770 16572 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.653s user 0.014s sys 0.015s
I20260810 18:51:41.113833 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:51:41.133801 16448 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.773s user 0.008s sys 0.025s
I20260810 18:51:41.134191 18551 leader_election.cc:288] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:51:41.117813 16783 debug-util.cc:398] Leaking SignalData structure 0x7b08002f6aa0 after lost signal to thread 16786
I20260810 18:51:41.117599 18499 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:41.059379 18510 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:41.142004 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.042814 18554 raft_consensus.cc:504] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:41.137193 18504 leader_election.cc:288] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:41.136636 18551 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
W20260810 18:51:41.156121 18495 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.629s user 0.008s sys 0.020s
I20260810 18:51:41.142253 18538 raft_consensus.cc:482] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
W20260810 18:51:41.175510 18495 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.649s user 0.009s sys 0.020s
I20260810 18:51:41.146159 18524 leader_election.cc:288] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:41.146039 18513 leader_election.cc:288] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:41.152750 18522 leader_election.cc:288] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:41.160703 18537 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:41.180626 18538 raft_consensus.cc:504] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:41.159808 18503 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.576s user 0.008s sys 0.028s
W20260810 18:51:41.200204 18503 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.620s user 0.008s sys 0.028s
W20260810 18:51:41.182242 16783 debug-util.cc:398] Leaking SignalData structure 0x7b080022a6c0 after lost signal to thread 17827
I20260810 18:51:41.166260 18499 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:41.185257 18520 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:41.136560 18560 raft_consensus.cc:504] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:41.240049 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:51:41.240324 16702 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.690s user 0.010s sys 0.013s
I20260810 18:51:41.176157 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.176153 18558 raft_consensus.cc:482] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:41.193226 18539 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:41.162899 18575 raft_consensus.cc:482] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:51:41.246249 16443 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.774s user 0.003s sys 0.036s
I20260810 18:51:41.180948 18551 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:41.259693 18552 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:41.260844 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.284119 18554 leader_election.cc:288] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:41.284044 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.261668 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.160985 18564 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:51:41.286759 16443 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.925s user 0.004s sys 0.037s
I20260810 18:51:41.277753 18531 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:41.262961 18505 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:41.263255 18538 leader_election.cc:288] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:41.266779 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.285907 18497 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:51:41.282586 16702 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.828s user 0.011s sys 0.014s
I20260810 18:51:41.303603 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.285315 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.292898 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.303112 18564 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:41.293274 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:41.287950 17727 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.032s sys 0.088s
I20260810 18:51:41.297484 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.285065 18558 raft_consensus.cc:504] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:51:41.299441 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.300527 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.299887 18579 raft_consensus.cc:482] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:41.290001 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.278290 18499 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:41.336405 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.314559 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.315250 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.318171 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.332923 18509 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:41.278196 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.318948 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.318315 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.292346 18510 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:41.318930 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.336184 18580 raft_consensus.cc:482] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:41.336582 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:51:41.333905 17727 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.730s user 0.032s sys 0.089s
I20260810 18:51:41.339013 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.344954 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:51:41.352352 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 315907us
I20260810 18:51:41.333910 18557 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:51:41.366711 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 369335us
I20260810 18:51:41.367262 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.337200 18527 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:41.391068 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.396332 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:41.412937 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.368165 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.417933 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:51:41.418540 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 321759us
I20260810 18:51:41.366276 18512 leader_election.cc:288] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:41.340037 18509 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:41.358289 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.432144 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.432915 18499 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:41.396893 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.415320 18557 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:41.443321 16695 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:41.289180 18575 raft_consensus.cc:504] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:41.448912 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.454090 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.450819 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.392297 18580 raft_consensus.cc:504] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:41.408861 18527 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:41.457065 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.457255 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.457610 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.458312 16745 raft_consensus.cc:2454] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 1.
I20260810 18:51:41.449316 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.438118 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:41.432202 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.449687 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.461802 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.333362 18579 raft_consensus.cc:504] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:41.463392 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.445451 18551 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:41.467418 16504 raft_consensus.cc:2454] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 1.
I20260810 18:51:41.468165 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.469514 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.469698 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.470176 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.471333 18560 leader_election.cc:288] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:41.479274 18558 leader_election.cc:288] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:41.471618 18529 raft_consensus.cc:2781] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:41.430099 18564 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:41.474385 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.477820 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:41.478178 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.485478 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:41.485601 17780 raft_consensus.cc:2454] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:41.448516 18557 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:51:41.490330 18590 log.cc:924] Time spent T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.134s user 0.000s sys 0.001s
I20260810 18:51:41.488787 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.497795 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.498325 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.499426 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:41.500015 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.501506 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.505716 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.506482 16745 raft_consensus.cc:2454] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:41.495877 18557 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:41.504336 18527 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:51:41.508025 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 250908us
I20260810 18:51:41.511198 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.512991 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.513981 16701 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:41.515751 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.516454 18516 raft_consensus.cc:2781] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:41.516762 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.516994 18516 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:41.517575 18516 raft_consensus.cc:3037] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:41.517491 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.519757 16701 leader_election.cc:302] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:41.523885 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.524160 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.525935 16752 raft_consensus.cc:2454] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 1.
I20260810 18:51:41.527794 18495 raft_consensus.cc:2781] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:41.528517 18495 raft_consensus.cc:482] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:41.529165 18495 raft_consensus.cc:3037] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:41.528904 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.498450 18579 leader_election.cc:288] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:41.524493 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.527846 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.524160 17727 leader_election.cc:302] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:41.484603 18529 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:41.537331 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.540177 18509 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:41.540362 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.540833 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.542079 17726 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:41.542941 18557 raft_consensus.cc:2726] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:41.544600 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.544938 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.545238 18529 raft_consensus.cc:3037] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:41.545724 18557 raft_consensus.cc:2781] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:41.546164 18557 raft_consensus.cc:482] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:41.546479 18557 raft_consensus.cc:3037] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:41.554216 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.554816 16745 raft_consensus.cc:2454] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:41.553478 18575 leader_election.cc:288] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:41.519466 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.556710 16701 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:41.560334 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.660619 18495 raft_consensus.cc:504] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:41.663370 18495 leader_election.cc:288] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:41.672087 18529 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:41.672690 18516 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:41.674574 18529 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:41.674007 18557 raft_consensus.cc:504] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:41.675923 18516 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:41.678030 18557 leader_election.cc:288] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:51:41.752961 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 196328us
I20260810 18:51:41.753108 18494 raft_consensus.cc:2781] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:41.753707 18494 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:41.754050 16701 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:41.754135 18494 raft_consensus.cc:3037] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:41.755180 18561 raft_consensus.cc:2781] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:41.755580 18561 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:41.755985 18561 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:41.760394 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.761463 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.762514 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.763401 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.764741 18561 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:41.765638 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.766258 16742 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:41.766814 18561 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:51:41.768486 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 247174us
I20260810 18:51:41.768715 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.769716 18503 raft_consensus.cc:482] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:41.769893 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.770232 18503 raft_consensus.cc:504] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:41.771093 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.763453 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.771977 16448 leader_election.cc:302] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:41.772181 18503 leader_election.cc:288] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:41.772140 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.772497 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.773444 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.773985 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.774535 18502 raft_consensus.cc:2726] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:41.774634 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.775547 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.775802 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.776180 16448 leader_election.cc:302] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:41.776816 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.777279 16448 leader_election.cc:302] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:41.774930 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.777832 18563 raft_consensus.cc:2726] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:41.777927 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.777588 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.777405 16747 raft_consensus.cc:2454] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:41.778970 18502 raft_consensus.cc:2726] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:41.788560 18494 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:41.790920 18494 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:41.793605 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.800661 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:51:41.819204 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 375952us
I20260810 18:51:41.821043 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.821357 16695 leader_election.cc:302] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:41.824759 18494 raft_consensus.cc:2781] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:41.825168 18494 raft_consensus.cc:482] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:41.825886 18494 raft_consensus.cc:3037] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:41.829648 16695 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:41.833630 18520 raft_consensus.cc:2726] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:41.833336 18494 raft_consensus.cc:504] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:41.833674 16695 leader_election.cc:302] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:41.836467 18522 raft_consensus.cc:2726] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:41.837155 16695 leader_election.cc:302] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:41.837888 18522 raft_consensus.cc:2726] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:41.838061 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.838315 16695 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:41.838804 16508 raft_consensus.cc:2454] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:41.839051 18522 raft_consensus.cc:2726] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:41.839497 16695 leader_election.cc:302] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:41.839682 18494 leader_election.cc:288] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:41.840394 18563 raft_consensus.cc:482] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:41.841063 16695 leader_election.cc:302] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:41.840999 18563 raft_consensus.cc:504] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:41.841843 18561 raft_consensus.cc:2726] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:41.842876 16695 leader_election.cc:302] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:41.843221 18563 leader_election.cc:288] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:41.844396 16695 leader_election.cc:302] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:41.845054 18561 raft_consensus.cc:2726] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:41.846036 16695 leader_election.cc:302] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:41.846786 18503 raft_consensus.cc:2726] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:41.845592 18522 raft_consensus.cc:2726] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:41.848119 16695 leader_election.cc:302] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:41.848806 18494 raft_consensus.cc:2726] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:41.849233 16695 leader_election.cc:302] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:41.849897 18494 raft_consensus.cc:2726] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:41.851043 16695 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:41.840435 18520 raft_consensus.cc:2726] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:41.851955 18522 raft_consensus.cc:2781] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:41.852366 18522 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:41.852735 18522 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:41.859974 18522 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:41.862042 18522 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:41.866894 18520 raft_consensus.cc:482] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:41.867491 18520 raft_consensus.cc:504] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:41.867966 18497 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:41.868641 18497 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:41.869522 18520 leader_election.cc:288] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:41.869717 16575 leader_election.cc:302] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:41.870051 18520 raft_consensus.cc:482] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:41.870640 18575 raft_consensus.cc:2781] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:41.870779 18497 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:41.871095 18520 raft_consensus.cc:504] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:41.871639 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:41.872821 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.873229 18520 leader_election.cc:288] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:41.871105 18575 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:41.874094 18575 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
W20260810 18:51:41.875810 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 444790us
I20260810 18:51:41.875978 16575 leader_election.cc:302] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:41.878211 18558 raft_consensus.cc:2726] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:41.882798 18580 leader_election.cc:288] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:41.884943 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.885713 16748 raft_consensus.cc:1226] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:41.885877 16752 raft_consensus.cc:2454] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:41.893273 18503 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:41.893949 18503 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:41.897709 18575 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:41.902259 16573 leader_election.cc:302] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:41.903632 18503 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:41.904247 18497 raft_consensus.cc:2726] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:41.915025 16575 leader_election.cc:302] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:41.906774 16573 leader_election.cc:302] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:41.917080 18603 raft_consensus.cc:2726] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:41.917222 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.917801 16573 leader_election.cc:302] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:41.918705 18603 raft_consensus.cc:2726] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:41.920542 18575 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:41.553247 17727 leader_election.cc:302] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:41.923532 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.924284 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.927352 18557 raft_consensus.cc:2781] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:41.927794 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.927878 18557 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:41.928553 18557 raft_consensus.cc:3037] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:41.929823 16573 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:41.931089 16573 leader_election.cc:302] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:41.931807 18594 raft_consensus.cc:2726] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:41.469523 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:41.932677 18594 raft_consensus.cc:2726] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:41.939760 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:41.922703 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 369579us
I20260810 18:51:41.946601 17727 leader_election.cc:302] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:41.916105 18497 raft_consensus.cc:2781] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:41.956715 18502 raft_consensus.cc:1050] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea: attempting to promote NON_VOTER 831925dd8b644ebab7eb0d672a94eddd to VOTER
I20260810 18:51:41.958155 18564 raft_consensus.cc:2726] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:41.959479 17727 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:41.960264 18564 raft_consensus.cc:2781] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:41.960786 18564 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:41.961201 18564 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:41.962463 16752 raft_consensus.cc:1226] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:41.963737 16748 raft_consensus.cc:1226] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:41.966145 16747 raft_consensus.cc:1226] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:41.967412 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.916208 16575 leader_election.cc:302] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:41.950436 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.959633 18502 consensus_queue.cc:237] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:41.956707 18497 raft_consensus.cc:482] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:42.029309 18497 raft_consensus.cc:3037] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:41.972057 18603 raft_consensus.cc:2726] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 18:51:41.972399 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 56274.2us
I20260810 18:51:42.034731 16622 raft_consensus.cc:1261] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:41.975699 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:41.978574 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:41.979779 16695 leader_election.cc:302] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:41.981777 17786 raft_consensus.cc:1226] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
W20260810 18:51:42.037694 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 58003.1us
I20260810 18:51:41.987910 16450 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:51:42.039269 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 51464.1us
I20260810 18:51:42.039377 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:42.039883 16450 leader_election.cc:302] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:42.041172 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:41.992463 16745 raft_consensus.cc:1226] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
W20260810 18:51:42.025115 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 53538.4us
I20260810 18:51:42.041846 16450 leader_election.cc:302] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:41.971451 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:42.043469 16623 raft_consensus.cc:1261] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:42.043741 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:42.044317 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:42.045732 16450 leader_election.cc:302] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:42.047001 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:42.047833 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:42.048301 16573 leader_election.cc:302] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:41.979311 17771 raft_consensus.cc:1261] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:42.052413 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:41.978876 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:42.053612 16623 raft_consensus.cc:1261] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:42.058005 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:41.992769 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:41.991971 17781 raft_consensus.cc:1261] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:41.986012 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:42.044615 18527 raft_consensus.cc:2726] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:42.059494 16450 leader_election.cc:302] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:41.988544 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:42.060016 18567 consensus_queue.cc:1035] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260810 18:51:42.054507 18551 raft_consensus.cc:2726] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:41.993184 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:41.980082 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:42.044538 18509 raft_consensus.cc:2726] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:42.046434 18520 raft_consensus.cc:2781] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:42.048010 18563 raft_consensus.cc:2726] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:42.054463 18558 raft_consensus.cc:2726] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:42.061219 18502 consensus_queue.cc:1035] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:42.067440 16450 leader_election.cc:302] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:42.071182 16450 leader_election.cc:302] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:42.072682 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:42.067824 17775 raft_consensus.cc:2454] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:42.066113 18503 consensus_queue.cc:1035] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260810 18:51:42.075006 16695 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:42.076076 18567 raft_consensus.cc:2781] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:42.076422 18567 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:42.076781 18567 raft_consensus.cc:3037] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:42.077803 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:42.078758 16450 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:42.079797 18509 raft_consensus.cc:2781] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:42.080205 18509 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:42.067081 18527 raft_consensus.cc:2726] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:42.080598 18509 raft_consensus.cc:3037] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:42.068904 18603 consensus_queue.cc:1046] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20260810 18:51:42.083427 18560 raft_consensus.cc:3032] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:42.083798 18560 raft_consensus.cc:729] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:42.084617 18560 consensus_queue.cc:260] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:42.068157 18520 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:42.089639 18560 raft_consensus.cc:3037] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:42.072219 18579 raft_consensus.cc:2781] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:42.081758 18502 raft_consensus.cc:2726] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:42.082238 16450 leader_election.cc:302] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:42.096086 18563 raft_consensus.cc:2726] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:42.094985 17726 leader_election.cc:302] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:42.095853 16450 leader_election.cc:302] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:42.091096 18520 raft_consensus.cc:3037] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:42.090577 18579 raft_consensus.cc:482] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:42.100787 16701 leader_election.cc:302] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:42.104122 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:42.104723 16749 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:42.099205 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:42.107674 18494 raft_consensus.cc:2726] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:42.114502 18503 consensus_queue.cc:1035] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:42.115485 18539 consensus_queue.cc:1046] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:42.103771 18551 consensus_queue.cc:1046] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20260810 18:51:42.117482 18527 raft_consensus.cc:3032] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:42.117846 18527 raft_consensus.cc:729] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:42.119344 18509 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:42.100597 18563 raft_consensus.cc:2781] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:42.120960 18516 consensus_queue.cc:1035] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:42.100023 18579 raft_consensus.cc:3037] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:42.097855 18502 consensus_queue.cc:1046] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260810 18:51:42.124805 18563 raft_consensus.cc:482] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:42.131701 18563 raft_consensus.cc:3037] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:42.133729 16701 leader_election.cc:302] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:42.135202 18602 raft_consensus.cc:2726] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 18:51:42.136662 16573 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.14.32.193:40791 timed out after 1.552s (SENT) [suppressed 3 similar messages]
I20260810 18:51:42.137076 18502 raft_consensus.cc:3032] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:42.141309 16450 leader_election.cc:302] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:42.146656 16450 leader_election.cc:302] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:42.138657 18502 raft_consensus.cc:729] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:42.150957 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:42.118709 18527 consensus_queue.cc:260] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:42.151530 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:42.153580 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:42.154698 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:42.156286 16749 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:42.156195 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:42.160241 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:42.144812 18538 consensus_queue.cc:1046] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20260810 18:51:42.167382 16749 raft_consensus.cc:1226] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:42.168929 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:42.149366 18539 raft_consensus.cc:2726] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:42.169855 16695 leader_election.cc:302] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:42.170569 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:42.172251 16695 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:42.144641 18564 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:42.195359 18522 raft_consensus.cc:2726] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:51:42.179503 18502 consensus_queue.cc:260] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:51:42.176966 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 82075.3us
I20260810 18:51:42.191984 18593 raft_consensus.cc:2726] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:51:42.192204 18527 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
W20260810 18:51:42.219283 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 45577.9us
I20260810 18:51:42.224609 16695 leader_election.cc:302] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:42.226493 16695 leader_election.cc:302] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:42.204244 18497 raft_consensus.cc:504] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:42.230005 18497 leader_election.cc:288] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:42.233042 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:42.210237 18538 raft_consensus.cc:2726] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:42.223186 18502 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:42.223919 18539 raft_consensus.cc:3032] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:42.205802 18614 raft_consensus.cc:2726] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:51:42.231710 18509 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:42.210446 18526 consensus_queue.cc:1046] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260810 18:51:42.238788 16626 raft_consensus.cc:1226] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:42.238967 16740 raft_consensus.cc:1261] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:42.239646 18539 raft_consensus.cc:729] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:42.260826 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:42.262332 16749 raft_consensus.cc:2379] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 2: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:51:42.243850 18564 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:42.249018 18538 consensus_queue.cc:1046] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260810 18:51:42.261191 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:42.285784 18567 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:42.260519 18557 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:42.274785 18593 raft_consensus.cc:2726] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 18:51:42.294082 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 62470.3us
W20260810 18:51:42.280572 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 49030.1us
I20260810 18:51:42.296784 16575 leader_election.cc:302] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:42.300241 18557 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:42.305262 17775 raft_consensus.cc:1226] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:42.304684 18618 raft_consensus.cc:2781] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:42.307559 18618 raft_consensus.cc:686] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
W20260810 18:51:42.308032 18607 log.cc:924] Time spent T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.144s user 0.002s sys 0.000s
I20260810 18:51:42.308477 18509 consensus_queue.cc:1035] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:42.308674 18618 consensus_queue.cc:237] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:42.306738 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:42.311899 18567 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:51:42.315590 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 131312us
I20260810 18:51:42.326575 17724 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:42.328864 16749 raft_consensus.cc:1226] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:42.304327 18538 raft_consensus.cc:3032] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:42.324694 16448 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:42.301069 18621 raft_consensus.cc:2726] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:51:42.335577 16745 raft_consensus.cc:1226] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:42.336020 18557 raft_consensus.cc:2726] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:42.334779 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:42.338930 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:42.356565 18579 raft_consensus.cc:504] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:42.347223 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:42.361615 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:42.340777 18538 raft_consensus.cc:729] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:42.392529 18520 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:42.400038 16749 raft_consensus.cc:1226] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:42.400636 18551 raft_consensus.cc:3032] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:42.401191 18551 raft_consensus.cc:729] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:42.301640 18539 consensus_queue.cc:260] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:42.409626 16573 leader_election.cc:302] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:42.358891 18579 leader_election.cc:288] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:42.388567 18563 raft_consensus.cc:504] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:42.420857 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:42.423504 16743 raft_consensus.cc:2379] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 2: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:51:42.427014 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:42.426563 18520 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:42.414258 18500 raft_consensus.cc:2726] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:42.433576 16747 raft_consensus.cc:1226] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:42.435199 18622 raft_consensus.cc:2726] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:51:42.438657 16506 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
I20260810 18:51:42.425060 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:42.460464 17724 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:42.467702 17724 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:42.469679 18557 raft_consensus.cc:2726] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:42.482183 16448 leader_election.cc:302] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:42.427047 18497 consensus_queue.cc:1046] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260810 18:51:42.416029 18594 consensus_queue.cc:1046] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20260810 18:51:42.508729 17724 leader_election.cc:302] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:42.513619 17724 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:42.443482 17726 leader_election.cc:302] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:42.516181 17724 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:42.500344 18622 raft_consensus.cc:2726] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:51:42.424467 18551 consensus_queue.cc:260] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:42.512444 18543 consensus_queue.cc:1046] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:42.510468 18564 raft_consensus.cc:2726] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:42.513022 18557 raft_consensus.cc:2726] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:42.518497 18614 raft_consensus.cc:2726] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 18:51:42.528364 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 103167us
I20260810 18:51:42.457902 18538 consensus_queue.cc:260] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:42.534940 17783 raft_consensus.cc:1261] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:51:42.543887 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:42.537299 18557 raft_consensus.cc:2781] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:42.544657 18557 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:42.545046 18557 raft_consensus.cc:3037] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:42.536069 18564 raft_consensus.cc:2726] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:51:42.549860 18543 raft_consensus.cc:3032] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:42.550325 18543 raft_consensus.cc:729] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:42.551244 18543 consensus_queue.cc:260] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:42.553422 18543 raft_consensus.cc:3037] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:42.563969 18563 leader_election.cc:288] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:42.587167 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:42.584939 18500 raft_consensus.cc:3032] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:42.606369 16450 leader_election.cc:302] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:51:42.612264 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 385816us
I20260810 18:51:42.613180 16695 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:42.616101 18622 raft_consensus.cc:2726] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:51:42.616806 18638 raft_consensus.cc:2726] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:51:42.617799 16695 leader_election.cc:302] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:51:42.617815 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 204666us
I20260810 18:51:42.619228 16573 leader_election.cc:302] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:42.634065 16695 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:42.605942 18500 raft_consensus.cc:729] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:42.613867 18563 raft_consensus.cc:482] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:42.620254 18594 raft_consensus.cc:2726] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 18:51:42.646937 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 217360us
I20260810 18:51:42.642930 18638 raft_consensus.cc:2726] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:51:42.654556 18567 raft_consensus.cc:2932] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:42.659412 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:42.659771 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:42.660351 16505 raft_consensus.cc:3037] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:42.640776 18602 raft_consensus.cc:2726] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:42.642189 18622 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:42.679231 18622 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:42.682219 18622 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:42.659432 18500 consensus_queue.cc:260] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:42.686659 18500 raft_consensus.cc:3037] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:42.695657 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:42.709803 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:42.650121 18563 raft_consensus.cc:504] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:42.717093 18563 leader_election.cc:288] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:42.717301 18538 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:42.718235 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:42.717725 18557 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:42.723738 18520 raft_consensus.cc:2932] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:42.725039 18557 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:42.727011 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:42.730365 18499 raft_consensus.cc:3032] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:42.737684 16505 raft_consensus.cc:2454] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
I20260810 18:51:42.738777 16752 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c"
dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
mode: GRACEFUL
new_leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
from {username='slave'} at 127.0.0.1:45870
I20260810 18:51:42.739542 16752 raft_consensus.cc:595] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Received request to transfer leadership to TS 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:42.739445 16448 leader_election.cc:302] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:42.740685 18499 raft_consensus.cc:729] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:42.718509 18502 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:42.740067 18622 consensus_queue.cc:1046] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:42.749372 18499 consensus_queue.cc:260] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:42.747012 17781 raft_consensus.cc:1226] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:42.765411 18622 raft_consensus.cc:3032] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:42.765818 18622 raft_consensus.cc:729] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:42.766628 18622 consensus_queue.cc:260] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:42.745373 17787 raft_consensus.cc:2932] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:42.736047 16507 raft_consensus.cc:2932] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:42.747681 18508 raft_consensus.cc:2726] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:42.773612 16747 raft_consensus.cc:1226] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:42.776624 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:42.787132 18622 raft_consensus.cc:3037] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:42.767988 18502 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:42.801765 18502 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:42.784024 18499 raft_consensus.cc:3037] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:42.818954 18626 raft_consensus.cc:3032] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:42.819784 18626 raft_consensus.cc:729] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:42.821221 18626 consensus_queue.cc:260] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:42.802606 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:42.831193 18560 raft_consensus.cc:482] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:42.831269 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:42.844122 18594 consensus_queue.cc:1046] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20260810 18:51:42.861354 18594 raft_consensus.cc:3032] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:42.862007 18594 raft_consensus.cc:729] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:42.846503 18648 consensus_queue.cc:1046] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260810 18:51:42.868639 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:42.851564 18641 raft_consensus.cc:979] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be: : Instructing follower 0397d760589d40d3b92ef141cdfc2dbb to start an election
W20260810 18:51:42.883523 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 367421us
I20260810 18:51:42.878578 18648 raft_consensus.cc:3032] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:42.892794 16450 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:42.856539 18560 raft_consensus.cc:504] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:42.879776 18634 raft_consensus.cc:482] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:42.904686 18634 raft_consensus.cc:504] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:42.913002 18634 leader_election.cc:288] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:42.879549 18594 consensus_queue.cc:260] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:42.894543 18648 raft_consensus.cc:729] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:42.922211 18628 raft_consensus.cc:1067] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Signalling peer 0397d760589d40d3b92ef141cdfc2dbb to start an election
I20260810 18:51:42.909303 18543 raft_consensus.cc:2726] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:42.937721 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:42.938347 16740 raft_consensus.cc:1261] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:51:42.938561 18648 consensus_queue.cc:260] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:42.946735 16626 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c"
dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
from {username='slave'} at 127.0.0.1:53878
I20260810 18:51:42.959869 16626 raft_consensus.cc:482] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:51:42.960458 16626 raft_consensus.cc:3037] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:42.869292 16495 raft_consensus.cc:2932] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:42.916960 18654 consensus_queue.cc:1046] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260810 18:51:42.968948 18654 raft_consensus.cc:3032] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:42.969485 18654 raft_consensus.cc:729] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:42.955268 18560 leader_election.cc:288] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:51:42.965860 17819 debug-util.cc:398] Leaking SignalData structure 0x7b08002dc900 after lost signal to thread 14470
I20260810 18:51:42.966329 18594 raft_consensus.cc:3037] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:42.977059 18626 raft_consensus.cc:3037] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:42.979593 16752 raft_consensus.cc:1226] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:42.993049 18551 raft_consensus.cc:3037] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:42.993824 18539 raft_consensus.cc:3037] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:42.957952 18563 raft_consensus.cc:2932] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:42.995105 16342 catalog_manager.cc:5495] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 1 to 2, leader changed from d304ac47fdf846de81337057a4ef7bea (127.14.32.193) to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 2 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:42.998342 16344 catalog_manager.cc:5495] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:42.985759 18648 raft_consensus.cc:3037] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:43.004681 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:42.978820 16740 raft_consensus.cc:2932] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:43.006969 18584 consensus_queue.cc:1035] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
W20260810 18:51:43.016770 17819 debug-util.cc:398] Leaking SignalData structure 0x7b08002cca20 after lost signal to thread 16391
I20260810 18:51:43.015097 18654 consensus_queue.cc:260] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:43.043279 16626 raft_consensus.cc:504] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:43.100956 16496 raft_consensus.cc:2454] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
I20260810 18:51:43.090893 18654 raft_consensus.cc:3037] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
W20260810 18:51:43.078487 17819 debug-util.cc:398] Leaking SignalData structure 0x7b08003510c0 after lost signal to thread 16545
W20260810 18:51:43.116097 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 385632us
I20260810 18:51:43.136490 16621 raft_consensus.cc:2932] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:43.156332 16695 leader_election.cc:302] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:43.160828 18602 raft_consensus.cc:2781] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:43.161634 18602 raft_consensus.cc:686] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:43.164049 18602 consensus_queue.cc:237] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:43.167282 17819 debug-util.cc:398] Leaking SignalData structure 0x7b0800351900 after lost signal to thread 16671
I20260810 18:51:43.159358 16695 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:43.170342 16695 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
W20260810 18:51:43.171882 17819 debug-util.cc:398] Leaking SignalData structure 0x7b0800355140 after lost signal to thread 16786
W20260810 18:51:43.173606 17819 debug-util.cc:398] Leaking SignalData structure 0x7b0800355a20 after lost signal to thread 17827
W20260810 18:51:43.174419 17819 debug-util.cc:398] Leaking SignalData structure 0x7b0800355be0 after lost signal to thread 18510
I20260810 18:51:43.190227 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:43.181526 18522 raft_consensus.cc:2781] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:43.200078 16695 leader_election.cc:302] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:43.182003 18503 raft_consensus.cc:2781] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:43.207834 18628 raft_consensus.cc:2726] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:43.207734 18522 raft_consensus.cc:686] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:43.233425 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:43.228971 18503 raft_consensus.cc:686] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:43.238466 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:43.238657 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:43.230933 16626 leader_election.cc:288] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:43.242645 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:43.244649 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:43.244915 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:43.245548 16501 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:43.248026 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:43.244275 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:43.254122 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:43.255973 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:43.259311 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:43.260012 16508 raft_consensus.cc:2454] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:43.260818 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:43.261834 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:43.263269 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:43.263836 16740 raft_consensus.cc:3032] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:43.264161 16740 raft_consensus.cc:729] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:43.264590 18503 consensus_queue.cc:237] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:43.264984 16740 consensus_queue.cc:260] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:43.256142 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:43.262964 16508 raft_consensus.cc:1226] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:43.311288 16505 raft_consensus.cc:2379] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 2: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:51:43.311491 16622 raft_consensus.cc:2932] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
W20260810 18:51:43.322021 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 65720.3us
I20260810 18:51:43.266599 16740 raft_consensus.cc:3037] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:43.356324 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:43.356793 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:43.360174 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:43.368780 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:43.371719 16508 raft_consensus.cc:1226] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:43.373095 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
W20260810 18:51:43.374262 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 61298.6us
I20260810 18:51:43.397814 16508 raft_consensus.cc:3037] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:43.340925 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:43.393934 16501 raft_consensus.cc:2454] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:43.438745 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:43.439232 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:43.439605 16504 raft_consensus.cc:2454] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:43.364135 18522 consensus_queue.cc:237] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:43.447683 18561 raft_consensus.cc:482] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:43.455703 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:43.456341 18561 raft_consensus.cc:504] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:43.461826 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:43.462831 16500 raft_consensus.cc:3037] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:43.463629 18561 leader_election.cc:288] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:51:43.465943 18669 log.cc:924] Time spent T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.130s user 0.000s sys 0.001s
I20260810 18:51:43.230463 17787 raft_consensus.cc:2932] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:43.468536 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:43.470808 16505 raft_consensus.cc:1226] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:43.358484 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:43.450371 16501 raft_consensus.cc:1226] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:43.480209 16447 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:43.487913 16447 leader_election.cc:302] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:43.490492 18580 raft_consensus.cc:2726] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:43.501487 16622 raft_consensus.cc:2932] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:43.505679 16447 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:43.508046 16447 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
W20260810 18:51:43.522330 18539 consensus_meta.cc:287] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.528s user 0.014s sys 0.000s
I20260810 18:51:43.500463 18652 raft_consensus.cc:2726] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:43.527669 16506 raft_consensus.cc:2454] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
W20260810 18:51:43.579227 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 52798us
W20260810 18:51:43.536670 18653 log.cc:924] Time spent T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260810 18:51:43.575155 17724 leader_election.cc:302] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:43.548214 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:43.527365 18543 raft_consensus.cc:2726] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:43.566959 18538 raft_consensus.cc:482] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:43.584120 18499 raft_consensus.cc:482] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:43.613507 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:43.625064 18656 raft_consensus.cc:2726] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:43.625126 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:43.612788 18538 raft_consensus.cc:504] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:43.633258 18666 log.cc:924] Time spent T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.174s user 0.000s sys 0.008s
I20260810 18:51:43.638300 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:43.641479 16447 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:43.641605 17724 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:43.576687 18662 consensus_queue.cc:1035] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20260810 18:51:43.639376 16636 raft_consensus.cc:2454] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:43.648782 16448 leader_election.cc:302] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:43.623597 17775 raft_consensus.cc:2379] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 2: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:51:43.646855 18656 raft_consensus.cc:2726] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:43.649525 18508 raft_consensus.cc:2726] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:43.661598 16508 raft_consensus.cc:2454] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:43.673442 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:43.679939 16749 raft_consensus.cc:2454] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:43.619483 18499 raft_consensus.cc:504] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:43.682119 16504 raft_consensus.cc:2454] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 2.
I20260810 18:51:43.656960 18622 raft_consensus.cc:2781] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:43.685707 17724 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:43.674326 18538 leader_election.cc:288] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:43.702399 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:43.703348 16631 raft_consensus.cc:2454] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
I20260810 18:51:43.693318 18622 raft_consensus.cc:482] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:43.696663 18656 raft_consensus.cc:2781] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:43.721801 16632 raft_consensus.cc:2454] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
W20260810 18:51:43.721921 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 76618.6us
I20260810 18:51:43.710892 18622 raft_consensus.cc:3037] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:43.727293 16502 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:43.728921 18499 leader_election.cc:288] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:43.731320 16740 raft_consensus.cc:2454] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:43.751492 16496 raft_consensus.cc:1261] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:51:43.738356 16701 leader_election.cc:302] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:43.740624 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:43.755760 16742 raft_consensus.cc:2454] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
W20260810 18:51:43.784793 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 46532.5us
I20260810 18:51:43.801292 18677 raft_consensus.cc:2781] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:43.801812 18677 raft_consensus.cc:482] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:43.802186 18677 raft_consensus.cc:3037] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:43.771562 16500 raft_consensus.cc:2454] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 2.
I20260810 18:51:43.808269 18502 raft_consensus.cc:2726] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:43.811403 17724 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:43.815021 16575 leader_election.cc:302] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:43.816190 16575 leader_election.cc:302] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:43.783708 18656 raft_consensus.cc:686] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:51:43.810873 18638 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:51:43.832530 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 230950us
I20260810 18:51:43.819504 18499 raft_consensus.cc:2781] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:43.819223 18594 raft_consensus.cc:2781] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:43.848112 16573 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:43.849454 16573 leader_election.cc:302] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:43.850637 18603 raft_consensus.cc:2726] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:43.851081 18499 raft_consensus.cc:686] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 2 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:43.852721 18499 consensus_queue.cc:237] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:43.846444 17775 raft_consensus.cc:2932] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:43.845364 18594 raft_consensus.cc:482] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:43.855059 18594 raft_consensus.cc:3037] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:43.861003 16573 leader_election.cc:302] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
W20260810 18:51:43.861693 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 50386.4us
I20260810 18:51:43.844882 18638 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:43.878031 18638 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:43.862172 18648 raft_consensus.cc:2726] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:43.873441 18526 raft_consensus.cc:2726] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 18:51:43.909608 18686 log.cc:924] Time spent T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.068s user 0.000s sys 0.001s
I20260810 18:51:43.892340 18699 raft_consensus.cc:2781] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:43.931413 18699 raft_consensus.cc:686] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:51:43.918447 18656 consensus_queue.cc:237] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:43.928517 18622 raft_consensus.cc:504] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:43.946564 18699 consensus_queue.cc:237] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:43.947757 18626 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:43.950649 18626 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:43.958105 18622 leader_election.cc:288] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:43.953063 18682 raft_consensus.cc:482] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:43.978890 18682 raft_consensus.cc:504] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:43.980115 18675 log.cc:924] Time spent T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.306s user 0.003s sys 0.000s
I20260810 18:51:43.958570 18594 raft_consensus.cc:504] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:43.979727 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:43.964201 18626 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:43.993726 18594 leader_election.cc:288] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:43.992975 18677 raft_consensus.cc:504] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:43.996786 16740 raft_consensus.cc:3037] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:43.997026 18677 leader_election.cc:288] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:43.982919 18687 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:44.029038 18687 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:44.031211 18682 leader_election.cc:288] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:51:44.026342 18697 log.cc:924] Time spent T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.182s user 0.000s sys 0.003s
I20260810 18:51:44.060285 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:44.061108 17787 raft_consensus.cc:2379] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 2: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:51:44.065379 16506 raft_consensus.cc:1261] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:51:44.070767 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:44.063367 18700 raft_consensus.cc:482] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:44.082475 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:44.083299 16622 raft_consensus.cc:3037] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 3
I20260810 18:51:44.069442 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:44.103745 16622 raft_consensus.cc:2454] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:51:44.110091 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:44.096997 18687 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:44.131438 18597 consensus_queue.cc:1035] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260810 18:51:44.170835 16448 leader_election.cc:302] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:51:44.181692 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 340590us
I20260810 18:51:44.184958 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:44.185638 16636 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
I20260810 18:51:44.180423 18652 raft_consensus.cc:2781] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:51:44.218228 18652 raft_consensus.cc:686] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:44.097236 18700 raft_consensus.cc:504] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:44.223114 18626 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:44.228000 16633 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:44.228102 18626 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:51:44.229333 16632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1125) took 1888 ms (client timeout 1617 ms). Trace:
I20260810 18:51:44.230968 18626 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:44.243505 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:44.241428 18700 leader_election.cc:288] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:44.245232 18700 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:44.249361 18702 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:44.223552 18593 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:44.180419 16740 raft_consensus.cc:2454] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:51:44.244421 16747 raft_consensus.cc:2454] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 2.
I20260810 18:51:44.248816 18700 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:44.220052 18652 consensus_queue.cc:237] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:44.270421 17727 leader_election.cc:302] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:44.273470 18707 raft_consensus.cc:2781] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:44.274101 18707 raft_consensus.cc:482] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:44.274642 18707 raft_consensus.cc:3037] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:44.252269 18593 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:44.250214 18702 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:44.297937 18702 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:44.304764 18700 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:44.341702 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:44.343401 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:44.344897 16623 raft_consensus.cc:2354] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3.
I20260810 18:51:44.345592 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:44.346731 16621 raft_consensus.cc:2379] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 3: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:44.353842 16622 raft_consensus.cc:1261] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:51:44.260732 16633 raft_consensus.cc:1261] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260810 18:51:44.230021 16632 rpcz_store.cc:269] 0810 18:51:42.341135 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:42.347962 (+ 6827us) service_pool.cc:227] Handling call
0810 18:51:42.868367 (+520405us) spinlock_profiling.cc:243] Waited 1 ms on lock 0x7b5800004298. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed03c1b0 00007f1aed03ba94 00007f1ae6723c04 00007f1ae6723b92
0810 18:51:44.229010 (+1360643us) spinlock_profiling.cc:243] Waited 372 ms on lock 0x7b540035eb54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:44.229170 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":857863936}
I20260810 18:51:44.390950 16496 raft_consensus.cc:1261] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260810 18:51:44.412554 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:44.413906 16740 raft_consensus.cc:2454] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 1.
I20260810 18:51:44.415287 18508 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:44.416585 18508 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:44.415481 18502 raft_consensus.cc:482] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:44.419035 18502 raft_consensus.cc:504] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:44.422098 18502 leader_election.cc:288] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:44.412778 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:44.424469 16628 raft_consensus.cc:2454] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 2.
I20260810 18:51:44.424988 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:44.451439 16631 raft_consensus.cc:2454] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:44.452016 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:44.453316 16742 raft_consensus.cc:2454] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:44.412263 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:44.463830 18593 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:44.484930 18529 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:44.485761 18529 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:44.487300 16448 leader_election.cc:302] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:44.487468 18561 raft_consensus.cc:482] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:44.488714 18561 raft_consensus.cc:504] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:44.514382 18622 raft_consensus.cc:2781] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:44.504271 18538 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:44.521029 17726 leader_election.cc:302] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:51:44.525774 18622 raft_consensus.cc:482] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260810 18:51:44.534368 16573 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.672s user 0.008s sys 0.058s
I20260810 18:51:44.540704 18622 raft_consensus.cc:3037] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
W20260810 18:51:44.551815 16573 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.689s user 0.009s sys 0.059s
W20260810 18:51:44.548882 18699 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.504s user 0.013s sys 0.052s
W20260810 18:51:44.547152 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 92383.3us
W20260810 18:51:44.554033 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 89054.6us
W20260810 18:51:44.573652 18499 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.039s sys 0.046s
I20260810 18:51:44.586320 18707 raft_consensus.cc:504] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:44.588045 18700 raft_consensus.cc:2781] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:44.588519 18700 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:44.588871 18700 raft_consensus.cc:3037] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:44.590231 18508 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:44.585024 18561 leader_election.cc:288] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:51:44.591643 18499 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.631s user 0.040s sys 0.046s
I20260810 18:51:44.597470 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:44.601760 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:44.585104 18538 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
W20260810 18:51:44.605405 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 744372us
I20260810 18:51:44.578909 18567 consensus_queue.cc:1035] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.076s
I20260810 18:51:44.606739 16573 leader_election.cc:302] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
W20260810 18:51:44.614148 18717 log.cc:924] Time spent T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.055s user 0.000s sys 0.002s
I20260810 18:51:44.596364 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:44.609233 18529 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:44.618185 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:44.618538 18707 leader_election.cc:288] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:44.618880 18499 raft_consensus.cc:2726] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:44.620636 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:44.628222 18716 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:44.629281 18716 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:44.602954 16496 raft_consensus.cc:2454] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
I20260810 18:51:44.609488 18538 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:44.632635 16342 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37026 (request call id 215) took 1748 ms. Trace:
I20260810 18:51:44.641745 18719 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:44.642575 18719 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:44.651120 18702 raft_consensus.cc:482] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:44.651146 16573 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:44.652062 18702 raft_consensus.cc:504] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:44.654129 18499 raft_consensus.cc:2726] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:44.656685 18702 leader_election.cc:288] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:44.658965 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:44.659657 17778 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 1 LEARNER]: Advancing to term 2
I20260810 18:51:44.659981 17775 raft_consensus.cc:2454] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:44.660295 16573 leader_election.cc:302] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
W20260810 18:51:44.660676 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 145424us
I20260810 18:51:44.661280 18499 raft_consensus.cc:2726] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:44.664080 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:44.664999 16747 raft_consensus.cc:3037] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:44.664938 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:44.669100 16448 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:44.670910 18535 raft_consensus.cc:2726] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:44.670991 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:44.672549 18707 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:44.673203 18707 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:44.674762 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:44.675587 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:44.675729 16628 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
W20260810 18:51:44.677567 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 371352us
I20260810 18:51:44.678324 16573 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:44.682262 18707 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:44.683465 18499 raft_consensus.cc:2726] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:44.648752 18622 raft_consensus.cc:504] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:44.685082 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:44.686774 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:44.692131 18622 leader_election.cc:288] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:44.693040 18719 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:51:44.645215 18656 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.006s sys 0.021s
I20260810 18:51:44.694278 16573 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
W20260810 18:51:44.694458 18656 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.742s user 0.007s sys 0.021s
W20260810 18:51:44.701095 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 62381.8us
I20260810 18:51:44.701287 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:44.703344 16631 raft_consensus.cc:3037] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:44.638089 16342 rpcz_store.cc:276] 0810 18:51:42.884375 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:42.884570 (+ 195us) service_pool.cc:227] Handling call
0810 18:51:44.632344 (+1747774us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 18:51:43.494685 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 18:51:43.498077 (+ 3392us) write_op.cc:432] Acquiring schema lock in shared mode
0810 18:51:43.498111 (+ 34us) write_op.cc:435] Acquired schema lock
0810 18:51:43.498151 (+ 40us) tablet.cc:660] Decoding operations
0810 18:51:43.498712 (+ 561us) write_op.cc:620] Acquiring the partition lock for write op
0810 18:51:43.498902 (+ 190us) write_op.cc:641] Partition lock acquired for write op
0810 18:51:43.498954 (+ 52us) tablet.cc:683] Acquiring locks for 1 operations
0810 18:51:43.499348 (+ 394us) tablet.cc:699] Row locks acquired
0810 18:51:43.499384 (+ 36us) write_op.cc:260] PREPARE: finished
0810 18:51:43.499555 (+ 171us) write_op.cc:270] Start()
0810 18:51:43.499733 (+ 178us) write_op.cc:276] Timestamp: P: 1786387903499509 usec, L: 0
0810 18:51:43.499756 (+ 23us) op_driver.cc:331] REPLICATION: starting
0810 18:51:43.584800 (+ 85044us) log.cc:842] Serialized 540 byte log entry
0810 18:51:44.220826 (+636026us) op_driver.cc:444] REPLICATION: finished
0810 18:51:44.567803 (+346977us) write_op.cc:301] APPLY: starting
0810 18:51:44.567946 (+ 143us) tablet.cc:1365] starting BulkCheckPresence
0810 18:51:44.568112 (+ 166us) tablet.cc:1368] finished BulkCheckPresence
0810 18:51:44.579900 (+ 11788us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 18:51:44.622290 (+ 42390us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 18:51:44.622590 (+ 300us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 18:51:44.622664 (+ 74us) write_op.cc:312] APPLY: finished
0810 18:51:44.623182 (+ 518us) log.cc:842] Serialized 22 byte log entry
0810 18:51:44.623484 (+ 302us) write_op.cc:489] Releasing partition, row and schema locks
0810 18:51:44.623622 (+ 138us) write_op.cc:454] Released schema lock
0810 18:51:44.623680 (+ 58us) write_op.cc:341] FINISH: Updating metrics
0810 18:51:44.624139 (+ 459us) write_op.cc:489] Releasing partition, row and schema locks
0810 18:51:44.624180 (+ 41us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":346517,"apply.run_cpu_time_us":3771,"apply.run_wall_time_us":56763,"num_ops":1,"prepare.queue_time_us":424998,"prepare.run_cpu_time_us":19882,"prepare.run_wall_time_us":270562,"raft.queue_time_us":892612,"raft.run_cpu_time_us":23148,"raft.run_wall_time_us":467484,"replication_time_us":720952,"thread_start_us":908690,"threads_started":4,"wal-append.queue_time_us":125454}]]}
I20260810 18:51:44.711436 16448 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:44.717272 18499 raft_consensus.cc:2726] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:44.718173 18535 raft_consensus.cc:2781] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:44.718662 18535 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:44.719040 18535 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:44.726073 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:44.726898 18539 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:44.728737 16623 raft_consensus.cc:2379] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 3: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:44.735423 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:44.738209 18731 raft_consensus.cc:482] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:44.724097 18707 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:44.740459 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:44.751475 18731 raft_consensus.cc:504] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:44.751616 18707 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:44.752720 16740 raft_consensus.cc:2379] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 2: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:51:44.740831 18716 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:44.745452 18539 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:44.755012 18539 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:44.755698 16344 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55260 (request call id 117) took 1795 ms. Trace:
I20260810 18:51:44.746447 18700 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:44.747545 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:44.761147 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:44.764621 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:44.748401 17778 raft_consensus.cc:1261] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEARNER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260810 18:51:44.758311 16623 raft_consensus.cc:1261] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260810 18:51:44.765909 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:44.766281 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:44.758666 16631 raft_consensus.cc:2454] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:51:44.759171 16345 catalog_manager.cc:5495] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea reported cstate change: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:44.756101 16344 rpcz_store.cc:276] 0810 18:51:42.959876 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:42.960071 (+ 195us) service_pool.cc:227] Handling call
0810 18:51:44.755485 (+1795414us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 18:51:43.722039 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 18:51:43.826009 (+103970us) write_op.cc:432] Acquiring schema lock in shared mode
0810 18:51:43.832593 (+ 6584us) write_op.cc:435] Acquired schema lock
0810 18:51:43.836632 (+ 4039us) tablet.cc:660] Decoding operations
0810 18:51:43.865230 (+ 28598us) write_op.cc:620] Acquiring the partition lock for write op
0810 18:51:43.868993 (+ 3763us) write_op.cc:641] Partition lock acquired for write op
0810 18:51:43.873300 (+ 4307us) tablet.cc:683] Acquiring locks for 1 operations
0810 18:51:43.901170 (+ 27870us) tablet.cc:699] Row locks acquired
0810 18:51:43.901202 (+ 32us) write_op.cc:260] PREPARE: finished
0810 18:51:43.922048 (+ 20846us) write_op.cc:270] Start()
0810 18:51:43.922258 (+ 210us) write_op.cc:276] Timestamp: P: 1786387903921972 usec, L: 0
0810 18:51:43.922282 (+ 24us) op_driver.cc:331] REPLICATION: starting
0810 18:51:43.923235 (+ 953us) log.cc:842] Serialized 598 byte log entry
0810 18:51:44.650066 (+726831us) op_driver.cc:444] REPLICATION: finished
0810 18:51:44.697625 (+ 47559us) write_op.cc:301] APPLY: starting
0810 18:51:44.699098 (+ 1473us) tablet.cc:1365] starting BulkCheckPresence
0810 18:51:44.701861 (+ 2763us) tablet.cc:1368] finished BulkCheckPresence
0810 18:51:44.701984 (+ 123us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 18:51:44.702286 (+ 302us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 18:51:44.702458 (+ 172us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 18:51:44.702506 (+ 48us) write_op.cc:312] APPLY: finished
0810 18:51:44.704555 (+ 2049us) log.cc:842] Serialized 22 byte log entry
0810 18:51:44.704746 (+ 191us) write_op.cc:489] Releasing partition, row and schema locks
0810 18:51:44.704831 (+ 85us) write_op.cc:454] Released schema lock
0810 18:51:44.704870 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0810 18:51:44.705187 (+ 317us) write_op.cc:489] Releasing partition, row and schema locks
0810 18:51:44.705220 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":109,"apply.run_cpu_time_us":2272,"apply.run_wall_time_us":7925,"num_ops":1,"prepare.queue_time_us":705132,"prepare.run_cpu_time_us":4934,"prepare.run_wall_time_us":201485,"replication_time_us":718015}]]}
I20260810 18:51:44.765542 16636 raft_consensus.cc:1261] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260810 18:51:44.758249 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:44.760258 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 66191.1us
I20260810 18:51:44.770282 16624 raft_consensus.cc:2379] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 2: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:51:44.772392 16448 leader_election.cc:302] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:44.774083 18539 raft_consensus.cc:2781] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:51:44.774871 18539 raft_consensus.cc:686] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:44.775729 16345 catalog_manager.cc:5495] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea reported cstate change: config changed from index 2 to 3, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:44.777024 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:44.772861 18622 consensus_queue.cc:1035] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260810 18:51:44.781611 17726 leader_election.cc:302] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:44.771413 16701 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:44.782574 17787 raft_consensus.cc:2354] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3.
I20260810 18:51:44.768574 18731 leader_election.cc:288] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:44.784099 16695 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:44.785070 16747 raft_consensus.cc:2454] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 3.
I20260810 18:51:44.800356 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:44.777814 18539 consensus_queue.cc:237] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:44.794663 18561 raft_consensus.cc:2781] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:44.801575 18561 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:44.801458 17727 leader_election.cc:302] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:44.801976 18561 raft_consensus.cc:3037] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:44.799317 18716 raft_consensus.cc:2726] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:44.803041 18716 consensus_queue.cc:1035] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260810 18:51:44.804710 18716 raft_consensus.cc:2781] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:51:44.805320 18716 raft_consensus.cc:686] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [term 3 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:51:44.784128 18656 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:44.793347 18529 raft_consensus.cc:2781] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:44.807216 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:44.807626 16745 raft_consensus.cc:1261] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260810 18:51:44.807976 16742 raft_consensus.cc:3037] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:44.808171 18656 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:44.801172 16750 raft_consensus.cc:3037] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 3
I20260810 18:51:44.807413 18529 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:44.818162 18529 raft_consensus.cc:3037] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:44.784727 18707 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:44.798525 18700 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:44.813742 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:44.817382 18520 consensus_queue.cc:1035] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:44.822466 16750 raft_consensus.cc:2454] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 3.
I20260810 18:51:44.838443 17727 leader_election.cc:302] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:44.838977 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:44.839563 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:44.840233 17787 raft_consensus.cc:2454] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:44.801808 17726 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:44.844774 18535 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:44.845822 17724 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:44.846733 18731 raft_consensus.cc:2781] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:51:44.806668 18716 consensus_queue.cc:237] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:44.882434 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:44.882681 18662 raft_consensus.cc:482] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:44.883401 18662 raft_consensus.cc:504] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:44.884438 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:44.884670 18535 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:44.885650 18662 leader_election.cc:288] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:44.847756 16695 leader_election.cc:398] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Vote denied by peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387) with higher term. Message: Invalid argument: T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3.
I20260810 18:51:44.886898 16695 leader_election.cc:401] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260810 18:51:44.887246 18580 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:44.888123 18580 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:44.887598 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 76623.3us
I20260810 18:51:44.890405 18580 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:44.891134 18728 raft_consensus.cc:482] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:44.891814 18728 raft_consensus.cc:504] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:44.894183 18728 leader_election.cc:288] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:44.895493 18567 raft_consensus.cc:2726] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387) with higher term. Message: Invalid argument: T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3.
I20260810 18:51:44.874234 16701 leader_election.cc:302] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:44.896672 18535 raft_consensus.cc:482] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:44.897395 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:44.898123 16631 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:44.899487 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:44.900959 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:44.901793 16636 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:44.875514 18707 raft_consensus.cc:3037] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:44.875386 18731 raft_consensus.cc:686] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:51:44.876168 18733 raft_consensus.cc:2726] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:44.879868 16573 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:44.880829 18499 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:44.905395 18499 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:44.880808 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:44.907490 16573 leader_election.cc:302] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:44.907516 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:44.908427 16450 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:44.909436 18709 raft_consensus.cc:2726] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:44.909536 18551 consensus_queue.cc:1035] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:51:44.849677 17726 leader_election.cc:302] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:44.897459 16701 leader_election.cc:398] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Vote denied by peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827) with higher term. Message: Invalid argument: T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3.
I20260810 18:51:44.913543 16701 leader_election.cc:401] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260810 18:51:44.922518 16448 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:44.921288 17724 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:44.932694 18499 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:44.934593 18707 raft_consensus.cc:2726] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:44.935195 16506 raft_consensus.cc:1226] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3. Ops: [2.3-2.3]
I20260810 18:51:44.922109 18561 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:44.935829 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:44.936218 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:44.937364 18561 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:44.938021 18516 raft_consensus.cc:2726] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827) with higher term. Message: Invalid argument: T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3.
I20260810 18:51:44.938810 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:44.940176 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:44.940912 16748 raft_consensus.cc:2354] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for earlier term 2. Current term is 3.
I20260810 18:51:44.940850 18522 consensus_queue.cc:1035] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:51:44.897573 18567 raft_consensus.cc:2726] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:44.943720 18503 consensus_queue.cc:1035] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:51:44.944342 16695 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
W20260810 18:51:44.952411 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 109444us
I20260810 18:51:44.953343 18702 raft_consensus.cc:2726] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:44.954068 18719 raft_consensus.cc:2726] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:44.954196 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:44.954841 16624 raft_consensus.cc:3037] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:44.956758 16631 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:51:44.957403 18505 raft_consensus.cc:2726] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:44.960251 18622 raft_consensus.cc:482] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:44.962205 16448 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:44.963703 17724 leader_election.cc:302] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:44.965749 17724 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:44.923707 18728 raft_consensus.cc:2726] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:51:44.971011 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:44.951589 18539 raft_consensus.cc:2781] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:44.968277 18719 raft_consensus.cc:2726] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:44.968371 18733 raft_consensus.cc:2726] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:44.960891 18622 raft_consensus.cc:504] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:44.979658 16742 raft_consensus.cc:2454] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:51:44.981354 18622 leader_election.cc:288] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:44.969481 18538 raft_consensus.cc:2781] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:51:44.981701 16573 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:44.981997 16624 raft_consensus.cc:2454] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 3.
I20260810 18:51:44.982534 18505 raft_consensus.cc:2726] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:44.987941 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:44.988705 16622 raft_consensus.cc:2354] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3.
I20260810 18:51:44.990749 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:44.991871 18731 consensus_queue.cc:237] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:44.996893 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:44.983999 16573 leader_election.cc:302] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:44.997570 16623 raft_consensus.cc:3037] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:44.897411 18535 raft_consensus.cc:504] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:45.000144 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:45.007354 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:45.008026 18622 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:45.009130 18538 raft_consensus.cc:686] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:44.972992 18516 raft_consensus.cc:2781] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:45.019260 16623 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:51:45.018772 16633 raft_consensus.cc:1226] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3. Ops: [2.3-2.3]
I20260810 18:51:45.020603 16450 leader_election.cc:302] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:45.021881 16450 leader_election.cc:302] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:44.974324 18529 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:45.014391 16344 catalog_manager.cc:5495] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 1 to 2, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 2 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:44.998307 18539 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:45.027652 16450 leader_election.cc:398] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Vote denied by peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171) with higher term. Message: Invalid argument: T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for earlier term 2. Current term is 3.
I20260810 18:51:45.029428 18622 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:45.027194 18539 raft_consensus.cc:3037] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:45.030256 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:45.032790 16573 leader_election.cc:302] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:45.016070 18505 raft_consensus.cc:2726] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:45.022773 18516 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:51:45.038170 16750 raft_consensus.cc:1226] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:45.031235 16344 catalog_manager.cc:5495] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } }
I20260810 18:51:45.037140 18730 raft_consensus.cc:2726] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:45.027016 18728 raft_consensus.cc:2726] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:45.019590 16621 raft_consensus.cc:2362] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2. Re-sending same reply.
I20260810 18:51:45.039713 18516 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
W20260810 18:51:45.044607 18737 log.cc:924] Time spent T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.120s user 0.001s sys 0.000s
I20260810 18:51:45.045024 18584 raft_consensus.cc:2781] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:45.049461 16450 leader_election.cc:302] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:45.052902 16631 raft_consensus.cc:3037] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:45.045071 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:45.034875 18656 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:45.068627 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:45.073156 17727 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:45.073518 16573 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:45.071201 18584 raft_consensus.cc:686] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:45.074987 16573 leader_election.cc:302] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:45.075553 18634 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:45.075495 18584 consensus_queue.cc:237] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:45.076237 18634 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:45.077003 18499 raft_consensus.cc:2726] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:45.065790 18728 raft_consensus.cc:2726] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:45.075526 18709 raft_consensus.cc:2781] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:45.081365 18622 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:45.104161 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:45.091677 18656 raft_consensus.cc:2726] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:45.110756 16450 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:45.112855 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:45.115288 16701 leader_election.cc:398] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Vote denied by peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827) with higher term. Message: Invalid argument: T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3.
I20260810 18:51:45.103359 18709 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:45.124102 18709 raft_consensus.cc:3037] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:45.117540 18535 leader_election.cc:288] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:45.117810 18631 raft_consensus.cc:482] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:45.117839 18551 raft_consensus.cc:482] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:45.133211 18551 raft_consensus.cc:504] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:45.113775 18538 consensus_queue.cc:237] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:45.143193 18663 consensus_queue.cc:1046] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260810 18:51:45.108381 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:45.121125 18505 consensus_queue.cc:1046] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:45.122248 18634 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:45.144344 18631 raft_consensus.cc:504] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:45.145049 18529 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:45.120047 18705 consensus_queue.cc:1046] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:45.157868 18705 raft_consensus.cc:3032] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:51:45.160683 18705 raft_consensus.cc:729] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:45.153429 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:45.131510 16701 leader_election.cc:302] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:51:45.163874 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 53202.6us
I20260810 18:51:45.170512 18730 raft_consensus.cc:2726] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 18:51:45.180617 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 49181.4us
I20260810 18:51:45.163020 18505 raft_consensus.cc:3032] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:45.169700 18539 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:45.204795 18602 raft_consensus.cc:2726] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:45.205294 18539 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:45.208498 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:45.161650 18705 consensus_queue.cc:260] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:45.212713 16636 raft_consensus.cc:2354] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for earlier term 2. Current term is 3.
I20260810 18:51:45.182405 18730 raft_consensus.cc:482] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
W20260810 18:51:45.219904 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 99407.6us
I20260810 18:51:45.203039 18535 raft_consensus.cc:3032] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:45.193929 18505 raft_consensus.cc:729] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:45.215481 16345 catalog_manager.cc:5495] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 1 to 2, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 2 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:51:45.228509 16448 leader_election.cc:398] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Vote denied by peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827) with higher term. Message: Invalid argument: T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for earlier term 2. Current term is 3.
I20260810 18:51:45.235380 16448 leader_election.cc:401] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20260810 18:51:45.237974 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 111386us
I20260810 18:51:45.239665 16345 catalog_manager.cc:5495] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 1 to 2, leader changed from d304ac47fdf846de81337057a4ef7bea (127.14.32.193) to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 2 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:51:45.244549 16631 raft_consensus.cc:1261] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:51:45.211606 18551 leader_election.cc:288] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:45.220768 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:45.231619 18535 raft_consensus.cc:729] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:45.254868 18535 consensus_queue.cc:260] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:45.255396 16573 leader_election.cc:302] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:45.242138 16345 catalog_manager.cc:5495] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 1 to 2, leader changed from d304ac47fdf846de81337057a4ef7bea (127.14.32.193) to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 2 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:45.257483 18579 raft_consensus.cc:3037] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:45.258966 18535 raft_consensus.cc:3037] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:45.263765 18631 leader_election.cc:288] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:45.256624 18730 raft_consensus.cc:504] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:45.264503 18505 consensus_queue.cc:260] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:45.266446 18505 raft_consensus.cc:3037] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:45.244410 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:45.268887 18730 leader_election.cc:288] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:45.270151 18584 raft_consensus.cc:2726] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:45.270864 18584 consensus_queue.cc:1046] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:45.272023 18762 raft_consensus.cc:3032] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:45.272148 18705 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:45.272488 18762 raft_consensus.cc:729] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:45.273993 16573 leader_election.cc:302] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:45.273556 18762 consensus_queue.cc:260] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:45.274814 18759 raft_consensus.cc:2741] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260810 18:51:45.219302 18709 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:45.275696 18762 raft_consensus.cc:3037] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:45.276644 18709 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:45.276779 18759 consensus_queue.cc:1035] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:45.279304 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:45.280776 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:45.281852 18579 raft_consensus.cc:2726] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827) with higher term. Message: Invalid argument: T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for earlier term 2. Current term is 3.
I20260810 18:51:45.286840 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:45.270648 18516 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:45.289232 18687 raft_consensus.cc:482] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:45.289378 16752 raft_consensus.cc:1226] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 2. Current term is 3. Ops: []
I20260810 18:51:45.289759 18687 raft_consensus.cc:504] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:45.291882 18687 leader_election.cc:288] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:45.292508 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:45.293126 16506 raft_consensus.cc:2454] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
I20260810 18:51:45.293556 18700 consensus_queue.cc:1046] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260810 18:51:45.296875 18516 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:45.297612 18656 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:45.297889 18700 raft_consensus.cc:3032] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:51:45.298563 18700 raft_consensus.cc:729] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:51:45.298247 18656 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:45.300753 18656 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:45.299939 18700 consensus_queue.cc:260] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:45.301981 18700 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:45.308401 16496 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:51:45.311201 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:45.311949 16506 raft_consensus.cc:2454] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 1.
I20260810 18:51:45.313048 17724 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:45.313915 18716 raft_consensus.cc:2781] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:45.314358 18716 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:45.314823 18716 raft_consensus.cc:3037] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Advancing to term 2
W20260810 18:51:45.318560 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 72588.4us
I20260810 18:51:45.325425 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:45.326624 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:45.327996 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:45.323868 16450 leader_election.cc:302] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:45.294400 18567 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:45.345201 17787 raft_consensus.cc:2454] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:51:45.348183 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:45.348785 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:45.348968 18567 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:45.349900 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 120259us
I20260810 18:51:45.351259 18567 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:45.337905 16495 raft_consensus.cc:2454] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
I20260810 18:51:45.348817 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:45.352593 16752 raft_consensus.cc:2354] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for earlier term 2. Current term is 3.
I20260810 18:51:45.353293 16695 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:45.353675 18538 raft_consensus.cc:2726] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:51:45.355053 18494 raft_consensus.cc:2781] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:51:45.355585 18494 raft_consensus.cc:686] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:45.356367 18494 consensus_queue.cc:237] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:45.390504 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:45.394101 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:45.394408 16495 raft_consensus.cc:2362] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2. Re-sending same reply.
I20260810 18:51:45.395107 18716 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:45.395653 16740 raft_consensus.cc:1261] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260810 18:51:45.397179 18539 raft_consensus.cc:482] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:45.397825 18579 raft_consensus.cc:482] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:45.397984 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:45.397771 18539 raft_consensus.cc:504] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:45.398546 18579 raft_consensus.cc:504] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:45.400031 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:45.400488 18579 leader_election.cc:288] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:45.401362 18716 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:45.401701 18539 leader_election.cc:288] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:45.402889 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:45.403036 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:45.404531 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:45.405447 16508 raft_consensus.cc:2379] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 2: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:45.405918 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:45.410246 16695 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:45.411150 18561 raft_consensus.cc:2781] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:45.411262 16695 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:45.411542 18561 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:45.411921 18561 raft_consensus.cc:3037] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:45.412273 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:45.413650 18705 raft_consensus.cc:2781] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:51:45.414228 18705 raft_consensus.cc:686] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:45.415405 18705 consensus_queue.cc:237] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:45.420478 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:45.421181 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:45.422086 16495 raft_consensus.cc:3037] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:45.421648 16344 catalog_manager.cc:5495] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 1 to 2, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196). New cstate: current_term: 2 leader_uuid: "831925dd8b644ebab7eb0d672a94eddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:45.424597 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:45.423964 16344 catalog_manager.cc:5495] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 1 to 3, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 3 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:45.424443 18561 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:45.426149 16344 catalog_manager.cc:5495] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 1 to 2, leader changed from d304ac47fdf846de81337057a4ef7bea (127.14.32.193) to 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196). New cstate: current_term: 2 leader_uuid: "831925dd8b644ebab7eb0d672a94eddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:45.427175 18561 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:45.435530 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:45.440547 16622 raft_consensus.cc:1261] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260810 18:51:45.444324 16496 raft_consensus.cc:2454] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:45.441049 16342 catalog_manager.cc:5495] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 1 to 2, leader changed from d304ac47fdf846de81337057a4ef7bea (127.14.32.193) to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 2 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:45.444796 16495 raft_consensus.cc:2454] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 2.
I20260810 18:51:45.446039 17724 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:45.447381 18716 raft_consensus.cc:2781] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:45.448031 18731 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:45.448733 18716 raft_consensus.cc:686] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:51:45.449723 18538 consensus_queue.cc:1035] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:45.450414 18716 consensus_queue.cc:237] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:45.451337 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:45.452140 16507 raft_consensus.cc:2379] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 2: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:45.448544 18731 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:45.454207 18731 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:45.455049 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:45.457579 16695 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:45.458468 18561 raft_consensus.cc:2781] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:45.459088 18561 raft_consensus.cc:686] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
W20260810 18:51:45.460139 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 63602.8us
I20260810 18:51:45.460005 18561 consensus_queue.cc:237] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:45.461071 16343 catalog_manager.cc:5495] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 1 to 3, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196). New cstate: current_term: 3 leader_uuid: "831925dd8b644ebab7eb0d672a94eddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:51:45.462195 18584 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:45.463011 18584 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:45.465075 18584 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:45.464094 16343 catalog_manager.cc:5495] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 1 to 3, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196). New cstate: current_term: 3 leader_uuid: "831925dd8b644ebab7eb0d672a94eddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:51:45.470062 16701 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:45.470184 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:45.470932 16495 raft_consensus.cc:2454] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 1.
I20260810 18:51:45.471343 18567 raft_consensus.cc:2781] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:45.473507 18567 raft_consensus.cc:686] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:45.474620 18567 consensus_queue.cc:237] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:45.478639 16742 raft_consensus.cc:1261] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260810 18:51:45.479312 18705 consensus_queue.cc:1035] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:51:45.487087 18656 consensus_queue.cc:1035] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260810 18:51:45.487501 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:45.488329 16752 raft_consensus.cc:2354] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 2. Current term is 3.
I20260810 18:51:45.496554 17724 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:45.497905 18656 raft_consensus.cc:2781] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:45.498329 18656 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:45.498741 18656 raft_consensus.cc:3037] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:45.498781 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:45.505190 16573 leader_election.cc:302] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:45.507275 18505 raft_consensus.cc:2726] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260810 18:51:45.510170 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:45.510988 17787 raft_consensus.cc:2454] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:45.510941 16622 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0"
dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
mode: GRACEFUL
new_leader_uuid: "831925dd8b644ebab7eb0d672a94eddd"
from {username='slave'} at 127.0.0.1:59734
I20260810 18:51:45.515607 16622 raft_consensus.cc:595] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Received request to transfer leadership to TS 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:45.516038 16622 raft_consensus.cc:603] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260810 18:51:45.516340 16573 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:45.517267 18505 raft_consensus.cc:2726] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:45.518379 16573 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:45.521446 18505 raft_consensus.cc:2726] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:45.521854 16496 raft_consensus.cc:1226] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:45.522557 16573 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:45.523286 18505 raft_consensus.cc:2726] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:51:45.510857 18656 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:45.526744 18656 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:45.527943 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:45.528721 16495 raft_consensus.cc:3037] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:45.534267 16573 leader_election.cc:302] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:45.534531 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:45.535039 18759 raft_consensus.cc:2726] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:51:45.540529 16573 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:45.541208 18759 raft_consensus.cc:2781] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:45.541669 18759 raft_consensus.cc:686] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:45.541765 16343 catalog_manager.cc:5495] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 2 to 3, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 3 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:45.542505 18759 consensus_queue.cc:237] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:45.548019 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 88956.9us
I20260810 18:51:45.553910 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:45.554339 16495 raft_consensus.cc:2454] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 2.
I20260810 18:51:45.555704 17724 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:45.556559 18656 raft_consensus.cc:2781] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:45.557219 18656 raft_consensus.cc:686] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:51:45.558288 18656 consensus_queue.cc:237] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:45.560709 16573 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:51:45.559878 18505 consensus_queue.cc:1046] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260810 18:51:45.561277 16450 leader_election.cc:302] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:45.561920 18505 raft_consensus.cc:3032] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:45.562351 18505 raft_consensus.cc:729] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:45.563411 18538 raft_consensus.cc:2726] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:45.563501 18505 consensus_queue.cc:260] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:45.564764 18505 raft_consensus.cc:3037] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:45.565852 16747 consensus_queue.cc:237] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:45.567474 18584 raft_consensus.cc:2781] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:45.567731 17787 raft_consensus.cc:1226] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:45.567876 18584 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:45.569149 18584 raft_consensus.cc:3037] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:45.569821 16622 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0"
dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
mode: GRACEFUL
new_leader_uuid: "831925dd8b644ebab7eb0d672a94eddd"
from {username='slave'} at 127.0.0.1:59734
I20260810 18:51:45.570418 16622 raft_consensus.cc:595] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Received request to transfer leadership to TS 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:45.570714 16622 raft_consensus.cc:603] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260810 18:51:45.574785 16342 catalog_manager.cc:5495] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 1 to 2, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196). New cstate: current_term: 2 leader_uuid: "831925dd8b644ebab7eb0d672a94eddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:51:45.576727 16631 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7dd134517a994065a45633249a3ce405"
dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
mode: GRACEFUL
new_leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
from {username='slave'} at 127.0.0.1:59734
I20260810 18:51:45.574729 16740 consensus_queue.cc:237] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:45.577266 16631 raft_consensus.cc:595] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Received request to transfer leadership to TS 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:45.577616 16631 raft_consensus.cc:603] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260810 18:51:45.583506 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:45.586020 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:45.586409 16631 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0"
dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
mode: GRACEFUL
new_leader_uuid: "831925dd8b644ebab7eb0d672a94eddd"
from {username='slave'} at 127.0.0.1:59734
I20260810 18:51:45.586956 16631 raft_consensus.cc:595] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Received request to transfer leadership to TS 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:45.597052 16345 catalog_manager.cc:5495] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 1 to 3, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 3 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:51:45.598958 16345 catalog_manager.cc:5495] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 2 to 3, leader changed from 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 3 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:45.608429 16342 catalog_manager.cc:5495] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 1 to 2, leader changed from d304ac47fdf846de81337057a4ef7bea (127.14.32.193) to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 2 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:45.610463 18584 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:45.615423 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:45.615959 17787 raft_consensus.cc:3037] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:45.562664 16450 leader_election.cc:302] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:45.622092 18538 raft_consensus.cc:2726] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:45.618417 18584 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:45.622632 18662 raft_consensus.cc:482] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:51:45.623413 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 60844.6us
I20260810 18:51:45.623152 18662 raft_consensus.cc:504] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:45.626135 18662 leader_election.cc:288] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:45.627595 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:45.628216 16631 raft_consensus.cc:2454] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
I20260810 18:51:45.629611 16495 raft_consensus.cc:1261] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:51:45.630096 17787 raft_consensus.cc:2454] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
I20260810 18:51:45.630578 16450 leader_election.cc:398] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Vote denied by peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171) with higher term. Message: Invalid argument: T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for earlier term 2. Current term is 3.
I20260810 18:51:45.631166 16573 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:51:45.632014 18584 consensus_queue.cc:1035] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:45.634485 18584 raft_consensus.cc:2781] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:51:45.638135 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:45.639159 18584 raft_consensus.cc:686] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 3 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:45.644910 18584 consensus_queue.cc:237] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:45.643009 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:45.653599 17787 raft_consensus.cc:1261] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:45.650786 16342 catalog_manager.cc:5495] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 1 to 2, leader changed from d304ac47fdf846de81337057a4ef7bea (127.14.32.193) to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 2 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:45.661705 18527 consensus_queue.cc:1035] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:51:45.662057 18499 raft_consensus.cc:979] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb: : Instructing follower 831925dd8b644ebab7eb0d672a94eddd to start an election
I20260810 18:51:45.663515 18505 raft_consensus.cc:1067] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Signalling peer 831925dd8b644ebab7eb0d672a94eddd to start an election
I20260810 18:51:45.664469 16342 catalog_manager.cc:5495] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 1 to 2, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 2 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:45.665446 16450 leader_election.cc:302] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:45.666661 16450 leader_election.cc:302] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:45.668545 18580 raft_consensus.cc:2726] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:45.669137 18580 raft_consensus.cc:2726] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:51:45.671860 16496 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061"
dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
mode: GRACEFUL
new_leader_uuid: "831925dd8b644ebab7eb0d672a94eddd"
from {username='slave'} at 127.0.0.1:51572
I20260810 18:51:45.672611 16496 raft_consensus.cc:595] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Received request to transfer leadership to TS 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:45.679358 17771 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0"
dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
from {username='slave'} at 127.0.0.1:55602
I20260810 18:51:45.679993 17771 raft_consensus.cc:482] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:51:45.680280 16344 catalog_manager.cc:5495] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 2 to 3, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 3 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260810 18:51:45.680332 17771 raft_consensus.cc:3037] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:45.686178 18567 consensus_queue.cc:1035] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:51:45.686764 17787 raft_consensus.cc:1261] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:45.687829 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:45.688391 16628 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d"
dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
mode: GRACEFUL
new_leader_uuid: "831925dd8b644ebab7eb0d672a94eddd"
from {username='slave'} at 127.0.0.1:59734
I20260810 18:51:45.699714 16628 raft_consensus.cc:595] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Received request to transfer leadership to TS 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:45.699893 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:45.700577 16622 raft_consensus.cc:2454] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 2.
I20260810 18:51:45.697634 16740 raft_consensus.cc:1261] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260810 18:51:45.702577 18656 consensus_queue.cc:1035] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260810 18:51:45.705892 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:45.708017 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:45.710204 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 7
I20260810 18:51:45.711714 18705 consensus_queue.cc:1035] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:51:45.714077 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:51:45.714972 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:51:45.720932 17771 raft_consensus.cc:504] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:45.722241 16345 catalog_manager.cc:5495] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 1 to 2, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196). New cstate: current_term: 2 leader_uuid: "831925dd8b644ebab7eb0d672a94eddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:51:45.728796 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:45.729799 16752 raft_consensus.cc:2454] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:45.755019 16622 raft_consensus.cc:1261] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260810 18:51:45.686803 16621 raft_consensus.cc:1261] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:45.774701 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:45.778271 16506 raft_consensus.cc:3037] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:45.785000 17771 leader_election.cc:288] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:45.786870 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:45.787738 16742 raft_consensus.cc:3037] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:45.788583 16621 raft_consensus.cc:1261] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:51:45.788771 16504 raft_consensus.cc:1261] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:45.790300 18527 consensus_queue.cc:1035] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:45.792356 16504 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:45.772181 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:45.811748 16504 raft_consensus.cc:1261] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260810 18:51:45.812737 16575 leader_election.cc:398] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Vote denied by peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171) with higher term. Message: Invalid argument: T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 2. Current term is 3.
I20260810 18:51:45.823549 16450 leader_election.cc:302] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:45.838572 16749 raft_consensus.cc:3037] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:45.838930 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:45.839422 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:45.840260 16621 raft_consensus.cc:3032] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:45.840652 16621 raft_consensus.cc:729] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:45.841717 16621 consensus_queue.cc:260] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:45.851630 16500 raft_consensus.cc:1226] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:45.856377 18584 consensus_queue.cc:1046] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260810 18:51:45.857348 18716 consensus_queue.cc:1035] T 91b6af1458cd44d8ba26b43579613ecb P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260810 18:51:45.857684 18584 raft_consensus.cc:3032] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:45.859117 18584 raft_consensus.cc:729] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:45.859632 16622 raft_consensus.cc:1261] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260810 18:51:45.861387 16500 raft_consensus.cc:1261] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260810 18:51:45.863811 18656 consensus_queue.cc:1035] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260810 18:51:45.870957 16506 raft_consensus.cc:2454] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:51:45.826506 16740 raft_consensus.cc:1261] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 3. (term mismatch)
I20260810 18:51:45.878515 16636 raft_consensus.cc:2454] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 2.
I20260810 18:51:45.833233 18728 raft_consensus.cc:2781] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:45.891260 16622 raft_consensus.cc:1261] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260810 18:51:45.886670 16740 pending_rounds.cc:76] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be: Aborting all ops after (but not including) 2
I20260810 18:51:45.888679 18656 consensus_queue.cc:1035] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260810 18:51:45.905860 18733 consensus_queue.cc:1035] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260810 18:51:45.883316 18561 consensus_queue.cc:1035] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20260810 18:51:45.897604 18728 raft_consensus.cc:482] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:45.924013 18728 raft_consensus.cc:3037] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:45.972429 16749 raft_consensus.cc:1261] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260810 18:51:45.978935 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 110353us
I20260810 18:51:45.908650 16740 pending_rounds.cc:97] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be: Aborting uncommitted NO_OP operation due to leader change: 2.3
I20260810 18:51:45.866717 16496 consensus_queue.cc:237] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:45.991269 16740 raft_consensus.cc:2866] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260810 18:51:45.997891 18804 log.cc:924] Time spent T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.090s user 0.000s sys 0.001s
W20260810 18:51:46.021005 18805 log.cc:924] Time spent T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.117s user 0.001s sys 0.000s
I20260810 18:51:45.897302 18584 consensus_queue.cc:260] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:51:46.048540 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 68663.2us
I20260810 18:51:46.049495 16695 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:46.051626 16742 raft_consensus.cc:2454] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 2.
I20260810 18:51:46.052001 18631 consensus_queue.cc:1035] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260810 18:51:46.052904 17778 raft_consensus.cc:1261] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260810 18:51:46.062585 18561 raft_consensus.cc:2781] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:51:46.065668 17787 raft_consensus.cc:1226] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3. Ops: []
W20260810 18:51:46.068418 18813 log.cc:924] Time spent T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.068s user 0.000s sys 0.001s
I20260810 18:51:46.068459 16505 raft_consensus.cc:1261] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260810 18:51:46.072219 17727 leader_election.cc:302] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:46.087342 18716 raft_consensus.cc:2781] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:46.087812 18561 raft_consensus.cc:686] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:46.088167 18716 raft_consensus.cc:686] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:51:46.091198 17778 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEARNER]: Advancing to term 3
I20260810 18:51:46.091454 18716 consensus_queue.cc:237] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:46.093984 16508 raft_consensus.cc:1261] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260810 18:51:46.102557 18561 consensus_queue.cc:237] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:46.076298 18535 consensus_queue.cc:1035] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260810 18:51:46.114892 16621 raft_consensus.cc:3037] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:46.118059 18584 raft_consensus.cc:3037] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:46.131973 18502 raft_consensus.cc:979] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea: : Instructing follower 831925dd8b644ebab7eb0d672a94eddd to start an election
I20260810 18:51:46.133105 18502 raft_consensus.cc:1067] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Signalling peer 831925dd8b644ebab7eb0d672a94eddd to start an election
I20260810 18:51:46.142988 18733 consensus_queue.cc:1035] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260810 18:51:46.145246 17778 raft_consensus.cc:1261] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 3 LEARNER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 3. (term mismatch)
I20260810 18:51:46.163044 17775 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061"
dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
from {username='slave'} at 127.0.0.1:55570
W20260810 18:51:46.172106 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 271689us
I20260810 18:51:46.188802 18687 raft_consensus.cc:482] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:46.189975 18687 raft_consensus.cc:504] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:46.275748 17775 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:51:46.290256 16701 leader_election.cc:302] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:46.198869 17778 pending_rounds.cc:76] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd: Aborting all ops after (but not including) 2
I20260810 18:51:46.295238 17778 pending_rounds.cc:97] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd: Aborting uncommitted NO_OP operation due to leader change: 2.3
I20260810 18:51:46.298964 17778 raft_consensus.cc:2866] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 3 LEARNER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:51:46.276700 17775 raft_consensus.cc:3037] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:46.304795 18567 raft_consensus.cc:2726] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:46.310847 16701 leader_election.cc:302] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:46.346215 16701 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:46.322093 18567 raft_consensus.cc:2726] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 18:51:46.362726 18810 log.cc:924] Time spent T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.087s user 0.002s sys 0.000s
I20260810 18:51:46.362668 18567 raft_consensus.cc:2781] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:46.339876 18527 consensus_queue.cc:1035] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20260810 18:51:46.369256 18687 leader_election.cc:288] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:46.403529 16631 raft_consensus.cc:1261] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260810 18:51:46.383893 18567 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:46.406723 18567 raft_consensus.cc:3037] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:46.440654 18631 consensus_queue.cc:1035] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260810 18:51:46.434168 18728 raft_consensus.cc:504] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:46.475812 17787 raft_consensus.cc:3037] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Advancing to term 2
W20260810 18:51:46.481650 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 135539us
I20260810 18:51:46.504137 17787 raft_consensus.cc:1261] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260810 18:51:46.467137 18809 consensus_queue.cc:1046] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260810 18:51:46.539808 17787 raft_consensus.cc:2932] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:46.552059 18848 raft_consensus.cc:3032] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:51:46.552485 18848 raft_consensus.cc:729] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:46.553282 18848 consensus_queue.cc:260] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:46.557857 18848 raft_consensus.cc:3037] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:46.560783 16742 raft_consensus.cc:1261] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260810 18:51:46.535948 18759 raft_consensus.cc:482] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:46.563851 18759 raft_consensus.cc:504] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:46.565938 18842 consensus_queue.cc:1035] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260810 18:51:46.566356 18759 leader_election.cc:288] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:51:46.571846 16573 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.026s sys 0.031s
W20260810 18:51:46.573503 16573 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.576s user 0.027s sys 0.031s
W20260810 18:51:46.573951 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 589098us
I20260810 18:51:46.537379 18819 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:46.570070 18700 consensus_queue.cc:1035] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
W20260810 18:51:46.598654 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 101459us
I20260810 18:51:46.598654 17787 raft_consensus.cc:1261] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260810 18:51:46.540553 16621 raft_consensus.cc:2454] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 2.
I20260810 18:51:46.608320 18819 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:46.611265 18819 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:46.613682 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:46.507516 18708 raft_consensus.cc:2932] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:46.618341 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:46.619189 16501 raft_consensus.cc:2454] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:46.618487 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:46.621718 16504 raft_consensus.cc:1226] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 2. Current term is 3. Ops: []
I20260810 18:51:46.628494 18567 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:46.630007 18567 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:46.615460 18819 consensus_queue.cc:1035] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260810 18:51:46.672003 18728 leader_election.cc:288] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:46.673100 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:46.462726 18561 raft_consensus.cc:2932] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:46.673811 16740 raft_consensus.cc:3037] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
W20260810 18:51:46.699378 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 192423us
I20260810 18:51:46.705154 16750 raft_consensus.cc:1261] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:46.693826 18708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:43720 (request call id 580) took 1116 ms. Trace:
I20260810 18:51:46.717756 16495 raft_consensus.cc:1226] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 2. Current term is 3. Ops: []
W20260810 18:51:46.740763 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 72379.1us
I20260810 18:51:46.747313 16573 leader_election.cc:302] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:46.788933 18817 raft_consensus.cc:2781] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:46.792627 18817 raft_consensus.cc:482] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:46.793044 18817 raft_consensus.cc:3037] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
W20260810 18:51:46.796474 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 84794.3us
W20260810 18:51:46.800053 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 196266us
W20260810 18:51:46.796078 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 289698us
I20260810 18:51:46.803674 16573 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
W20260810 18:51:46.790627 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 60054.4us
I20260810 18:51:46.831812 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:46.850490 16505 raft_consensus.cc:2379] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 2: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:46.817168 18819 raft_consensus.cc:2726] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:46.799011 17775 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:46.816131 18728 consensus_queue.cc:1035] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.055s
I20260810 18:51:46.816334 18808 consensus_queue.cc:1035] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20260810 18:51:46.875686 16633 raft_consensus.cc:1261] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260810 18:51:46.874758 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:46.736580 18708 rpcz_store.cc:276] 0810 18:51:45.548732 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:45.563551 (+ 14819us) service_pool.cc:227] Handling call
0810 18:51:45.569113 (+ 5562us) log.cc:842] Serialized 501 byte log entry
0810 18:51:46.664936 (+1095823us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":966,"raft.run_cpu_time_us":969,"raft.run_wall_time_us":968,"thread_start_us":63319,"threads_started":1,"wal-append.queue_time_us":63495}
I20260810 18:51:46.858242 18584 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:46.882161 18584 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:46.884702 18584 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:46.840071 18505 consensus_queue.cc:1046] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260810 18:51:46.891738 18505 raft_consensus.cc:3032] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:51:46.892174 18505 raft_consensus.cc:729] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:46.893172 18505 consensus_queue.cc:260] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:46.894623 18505 raft_consensus.cc:3037] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
W20260810 18:51:46.898306 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 95381.8us
I20260810 18:51:46.919131 16631 raft_consensus.cc:1261] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260810 18:51:46.919153 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:46.906152 18861 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:46.930166 16750 raft_consensus.cc:1261] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260810 18:51:46.930677 16750 pending_rounds.cc:76] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be: Aborting all ops after (but not including) 4
I20260810 18:51:46.930899 16750 pending_rounds.cc:97] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260810 18:51:46.931134 16750 raft_consensus.cc:2866] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260810 18:51:46.932545 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 67972.7us
I20260810 18:51:46.935709 16752 raft_consensus.cc:1226] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 2. Current term is 3. Ops: []
I20260810 18:51:46.937960 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:46.938776 16752 raft_consensus.cc:2454] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:46.939940 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:46.941520 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:46.948953 17771 raft_consensus.cc:2932] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:46.959283 16328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 7ef7c65533724e678b75830002906e2c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 18:51:46.916978 17778 raft_consensus.cc:1226] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:46.924006 18848 consensus_queue.cc:1035] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
W20260810 18:51:46.992192 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 132984us
I20260810 18:51:47.003108 18733 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:47.004175 18733 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:47.016320 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:47.020273 16496 raft_consensus.cc:1261] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260810 18:51:47.022214 18709 consensus_queue.cc:1035] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260810 18:51:46.962955 18861 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:47.034193 18861 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:47.027246 16621 raft_consensus.cc:1226] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 3 LEADER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3. Ops: []
I20260810 18:51:47.045492 16742 raft_consensus.cc:1226] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
W20260810 18:51:47.090188 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 51357.8us
I20260810 18:51:47.085897 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:47.093015 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:47.104382 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:47.108318 16502 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
I20260810 18:51:46.995432 18871 consensus_queue.cc:1035] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.110s
I20260810 18:51:47.100807 17775 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:47.104087 18733 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:47.083892 18899 consensus_queue.cc:1046] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.088s
I20260810 18:51:47.101274 18883 consensus_queue.cc:1046] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
W20260810 18:51:47.243963 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 249597us
I20260810 18:51:47.175231 18819 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:47.190423 17775 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:55570 (request call id 296) took 1010 ms. Trace:
I20260810 18:51:47.273185 17775 rpcz_store.cc:276] 0810 18:51:46.161579 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:46.162086 (+ 507us) service_pool.cc:227] Handling call
0810 18:51:47.171640 (+1009554us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:51:47.273379 18866 consensus_queue.cc:1046] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:51:47.274571 18866 consensus_queue.cc:1225] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer 980f343dd1fd4e049143f8b9d9f2f1be log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260810 18:51:47.256912 18906 consensus_queue.cc:1035] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260810 18:51:47.258061 18817 raft_consensus.cc:504] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:47.258807 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:47.271289 18905 raft_consensus.cc:3032] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 LEADER]: Stepping down as leader of term 2
W20260810 18:51:47.231628 18889 log.cc:924] Time spent T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.127s user 0.000s sys 0.000s
I20260810 18:51:47.253115 18899 raft_consensus.cc:482] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:47.252820 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:47.263412 18913 raft_consensus.cc:482] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:47.211426 18733 consensus_queue.cc:1035] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.096s
I20260810 18:51:47.279188 18843 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:47.264406 18910 raft_consensus.cc:482] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:47.289868 18899 raft_consensus.cc:504] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:47.280893 18819 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:47.293654 16623 raft_consensus.cc:2354] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3.
W20260810 18:51:47.289052 16740 consensus_meta.cc:287] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.588s user 0.011s sys 0.000s
I20260810 18:51:47.279999 18866 raft_consensus.cc:482] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:47.282789 18905 raft_consensus.cc:729] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:47.327593 16504 raft_consensus.cc:3032] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:47.298970 18561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:43720 (request call id 581) took 1717 ms. Trace:
I20260810 18:51:47.355345 16504 raft_consensus.cc:729] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:47.337383 18862 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:47.332399 18883 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:47.292768 17727 leader_election.cc:302] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:47.330019 18913 raft_consensus.cc:504] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:47.330595 18843 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:47.426780 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:47.331852 16740 raft_consensus.cc:2454] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:51:47.487277 16622 raft_consensus.cc:3037] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:47.347894 18662 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:47.427490 16504 consensus_queue.cc:260] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:51:47.480795 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 238662us
I20260810 18:51:47.495604 18902 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:47.348068 18527 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:47.496173 18902 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:47.496793 18527 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:47.347337 18535 raft_consensus.cc:482] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
W20260810 18:51:47.456065 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 48236us
I20260810 18:51:47.337152 18866 raft_consensus.cc:504] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:47.426436 18508 raft_consensus.cc:482] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:47.498795 18902 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:47.332857 18910 raft_consensus.cc:504] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:47.499058 18527 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:47.606473 18508 raft_consensus.cc:504] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:47.596115 18502 raft_consensus.cc:482] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:47.608659 18502 raft_consensus.cc:504] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:47.610591 18505 consensus_meta.cc:287] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.712s user 0.008s sys 0.004s
I20260810 18:51:47.425014 18584 raft_consensus.cc:482] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
W20260810 18:51:47.602391 18849 log.cc:924] Time spent T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260810 18:51:47.501564 18662 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:47.529894 18656 raft_consensus.cc:2726] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:47.602589 18870 raft_consensus.cc:482] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:47.445394 18862 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:47.613615 18584 raft_consensus.cc:504] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:47.475221 16626 raft_consensus.cc:2454] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
I20260810 18:51:47.627868 18899 leader_election.cc:288] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:47.607052 18527 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:47.648730 18584 leader_election.cc:288] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:47.642263 18508 leader_election.cc:288] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:47.649303 18527 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:47.650501 18859 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:47.571110 18913 leader_election.cc:288] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:47.651405 18859 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:47.654461 18859 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:47.656430 18888 raft_consensus.cc:482] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:47.657385 18888 raft_consensus.cc:504] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:47.658356 17778 raft_consensus.cc:1261] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 18:51:47.599181 18730 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:47.611577 18505 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:47.547686 18538 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:47.619450 18535 raft_consensus.cc:504] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:47.548729 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 257729us
I20260810 18:51:47.644068 18923 raft_consensus.cc:482] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:47.640276 18906 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:47.645588 18502 leader_election.cc:288] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:47.357996 18561 rpcz_store.cc:276] 0810 18:51:45.570220 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:45.571310 (+ 1090us) service_pool.cc:227] Handling call
0810 18:51:45.625011 (+ 53701us) log.cc:842] Serialized 501 byte log entry
0810 18:51:47.287605 (+1662594us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":12975,"raft.run_cpu_time_us":1045,"raft.run_wall_time_us":1924,"thread_start_us":37980,"threads_started":1,"wal-append.queue_time_us":10899,"wal-append.run_cpu_time_us":1634,"wal-append.run_wall_time_us":1002754}
I20260810 18:51:47.571429 18817 leader_election.cc:288] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:47.475430 18905 consensus_queue.cc:260] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:47.649839 18924 consensus_queue.cc:1035] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260810 18:51:47.633802 18870 raft_consensus.cc:504] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:47.674086 18926 raft_consensus.cc:482] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:47.723058 18906 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:47.753996 18870 leader_election.cc:288] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:47.669087 18828 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:47.712611 18923 raft_consensus.cc:504] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:47.739851 18819 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:47.670521 18730 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:47.736491 18926 raft_consensus.cc:504] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:47.675386 18527 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:47.671209 18505 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:47.516845 18883 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:47.696933 18538 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:47.774560 18923 leader_election.cc:288] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:47.796332 18906 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:47.771759 18828 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:47.800832 18843 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:47.797772 16501 raft_consensus.cc:2932] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:47.845633 18505 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:47.799399 18730 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:47.841934 18828 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:47.780320 18888 leader_election.cc:288] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:47.859021 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:47.825767 18905 raft_consensus.cc:3037] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:47.881633 16622 raft_consensus.cc:2454] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 2.
I20260810 18:51:47.817242 18862 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:47.772759 18866 leader_election.cc:288] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:47.875483 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:47.889559 16505 raft_consensus.cc:1261] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260810 18:51:47.850468 18538 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:47.891710 16502 raft_consensus.cc:2379] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 2: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:47.905508 18910 leader_election.cc:288] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:47.938532 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:47.943820 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:47.946257 16342 catalog_manager.cc:5495] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 1 to 2, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196). New cstate: current_term: 2 leader_uuid: "831925dd8b644ebab7eb0d672a94eddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:51:47.950574 16344 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37026 (request call id 224) took 1256 ms. Trace:
I20260810 18:51:47.951025 16344 rpcz_store.cc:276] 0810 18:51:46.694205 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:46.694478 (+ 273us) service_pool.cc:227] Handling call
0810 18:51:47.950501 (+1256023us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":4}
I20260810 18:51:47.953410 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:47.954864 16345 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37030 (request call id 201) took 1031 ms. Trace:
I20260810 18:51:47.957965 16345 rpcz_store.cc:276] 0810 18:51:46.923323 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:46.927533 (+ 4210us) service_pool.cc:227] Handling call
0810 18:51:47.954789 (+1027256us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":965002,"reported_tablets":1}
I20260810 18:51:47.938283 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:47.959653 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:47.963696 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:47.963584 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:47.964551 16749 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:48.011754 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:48.012493 16450 leader_election.cc:302] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:48.048995 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:51:48.013342 18535 leader_election.cc:288] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:48.043618 18926 leader_election.cc:288] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:48.068390 18848 raft_consensus.cc:482] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:51:48.012924 16740 raft_consensus.cc:2454] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:48.069236 18848 raft_consensus.cc:504] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:51:48.096849 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 72910.2us
I20260810 18:51:48.119125 16450 leader_election.cc:302] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:48.064781 18859 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:48.152253 16450 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:48.154567 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:48.110224 18662 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:48.160893 18874 raft_consensus.cc:2781] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:48.161705 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:48.164322 18563 raft_consensus.cc:2781] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:51:48.165179 18882 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:48.117699 18502 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:48.120453 18852 raft_consensus.cc:482] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:48.153198 18700 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:48.184547 18662 raft_consensus.cc:2781] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:48.185583 18874 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
W20260810 18:51:48.215899 16624 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.715s user 0.000s sys 0.036s
I20260810 18:51:48.173972 18859 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:51:48.230271 16624 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.718s user 0.000s sys 0.036s
I20260810 18:51:48.231698 16507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56468 (request call id 67) took 1125 ms. Trace:
I20260810 18:51:48.232988 16507 rpcz_store.cc:276] 0810 18:51:47.089600 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:47.089784 (+ 184us) service_pool.cc:227] Handling call
0810 18:51:48.211154 (+1121370us) spinlock_profiling.cc:243] Waited 1.6 ms on lock 0x7b5800001898. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed0502b0 00007f1aed03b7f9 00007f1aed03b2a2 00007f1ae6721684
0810 18:51:48.214221 (+ 3067us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:48.214434 (+ 213us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:48.214469 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:48.214523 (+ 54us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:51:48.214564 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:48.214633 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:48.214746 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3688704}
I20260810 18:51:48.228263 18656 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:48.234674 18656 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:48.235047 16624 raft_consensus.cc:2932] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:48.237600 18656 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:48.209558 18848 leader_election.cc:288] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:51:48.251657 16575 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.787s user 0.013s sys 0.029s
I20260810 18:51:48.194366 18882 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:48.211766 18563 raft_consensus.cc:686] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:48.191589 18502 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:48.194926 18700 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:48.212229 18662 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:48.215307 18874 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:48.268769 18882 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:51:48.254175 16575 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.789s user 0.013s sys 0.029s
I20260810 18:51:48.212961 18733 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:48.275573 18733 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:48.278194 18733 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:48.284920 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:48.287451 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:48.288337 16746 raft_consensus.cc:3037] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 3
I20260810 18:51:48.241087 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:51:48.257714 16573 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.824s user 0.040s sys 0.027s
W20260810 18:51:48.248060 16632 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.757s user 0.005s sys 0.031s
W20260810 18:51:48.300969 16573 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.872s user 0.040s sys 0.027s
W20260810 18:51:48.301432 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 872308us
W20260810 18:51:48.304281 17780 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.887s user 0.025s sys 0.038s
W20260810 18:51:48.304718 17780 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.957s user 0.025s sys 0.038s
I20260810 18:51:48.333410 16701 leader_election.cc:398] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Vote denied by peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827) with higher term. Message: Invalid argument: T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3.
I20260810 18:51:48.334054 16701 leader_election.cc:401] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260810 18:51:48.335304 16504 raft_consensus.cc:3037] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:48.257717 18883 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:48.266004 18662 raft_consensus.cc:3037] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
W20260810 18:51:48.307512 16632 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.817s user 0.006s sys 0.032s
I20260810 18:51:48.196386 18852 raft_consensus.cc:504] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:48.344760 16701 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:48.359431 18859 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:48.359954 18656 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:48.381385 18656 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:48.381920 18700 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:48.337467 18882 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:48.394997 16573 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:48.416380 18656 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:48.416720 18502 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:48.418862 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:48.423593 18563 consensus_queue.cc:237] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:48.424263 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:48.430297 16504 raft_consensus.cc:2454] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 2.
I20260810 18:51:48.371285 18871 raft_consensus.cc:2781] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:48.441833 16504 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56468 (request call id 65) took 1363 ms. Trace:
I20260810 18:51:48.444242 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:48.460063 18874 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:48.446805 18728 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260810 18:51:48.470248 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:51:48.467317 16447 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.894s user 0.000s sys 0.055s
W20260810 18:51:48.471424 16447 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.898s user 0.000s sys 0.055s
W20260810 18:51:48.472194 18933 log.cc:924] Time spent T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.254s user 0.000s sys 0.001s
I20260810 18:51:48.475134 18852 leader_election.cc:288] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:51:48.476119 18905 consensus_meta.cc:287] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.592s user 0.007s sys 0.007s
I20260810 18:51:48.442601 16504 rpcz_store.cc:276] 0810 18:51:47.068143 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:47.069830 (+ 1687us) service_pool.cc:227] Handling call
0810 18:51:47.327478 (+257648us) spinlock_profiling.cc:243] Waited 3.66 ms on lock 0x7b5400052420. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae808a730 00007f1ae80fdeff 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86
0810 18:51:48.305610 (+978132us) spinlock_profiling.cc:243] Waited 809 ms on lock 0x7b58000c41b0. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae8075f63 00007f1ae80d9aba 00007f1ae80f13af 00007f1ae80f2b4e 00007f1ae80fe228 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c
0810 18:51:48.430984 (+125374us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1869464192}
I20260810 18:51:48.448655 16632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60066 (request call id 70) took 1077 ms. Trace:
I20260810 18:51:48.465308 18871 raft_consensus.cc:686] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:48.411517 18882 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:48.487481 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:48.489470 17787 raft_consensus.cc:2454] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
W20260810 18:51:48.488016 16448 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.920s user 0.026s sys 0.030s
I20260810 18:51:48.466133 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:48.467845 18516 raft_consensus.cc:482] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:48.468787 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:48.483855 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:51:48.492130 16448 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.924s user 0.026s sys 0.030s
I20260810 18:51:48.618089 16624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1231) took 1247 ms. Trace:
I20260810 18:51:48.557756 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:48.565397 16572 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.880s user 0.007s sys 0.030s
W20260810 18:51:48.645167 16752 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.023s user 0.022s sys 0.031s
W20260810 18:51:48.714341 16572 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.200s user 0.007s sys 0.037s
W20260810 18:51:48.714682 16752 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.098s user 0.023s sys 0.032s
W20260810 18:51:48.645695 16750 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.689s user 0.000s sys 0.049s
W20260810 18:51:48.715931 16750 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.759s user 0.002s sys 0.050s
I20260810 18:51:48.509557 16746 raft_consensus.cc:2454] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
I20260810 18:51:48.711433 16505 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1243) took 1103 ms. Trace:
I20260810 18:51:48.710309 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:48.713564 16631 raft_consensus.cc:2454] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 1.
W20260810 18:51:48.596668 17724 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.14.32.193:40791 timed out after 1.555s (SENT) [suppressed 6 similar messages]
W20260810 18:51:48.485837 17771 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.874s user 0.023s sys 0.033s
W20260810 18:51:48.645404 16342 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.681s user 0.026s sys 0.021s
W20260810 18:51:48.492527 16693 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.852s user 0.063s sys 0.084s
W20260810 18:51:48.488131 16621 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.810s user 0.021s sys 0.037s
I20260810 18:51:48.596131 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:48.717446 16505 rpcz_store.cc:276] 0810 18:51:47.607681 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:47.900684 (+293003us) service_pool.cc:227] Handling call
0810 18:51:48.626872 (+726188us) spinlock_profiling.cc:243] Waited 1.35 ms on lock 0x7b5800001898. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed0502b0 00007f1aed03b7f9 00007f1aed03b2a2 00007f1ae6721684
0810 18:51:48.627142 (+ 270us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:48.627411 (+ 269us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:48.627451 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:48.627504 (+ 53us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:51:48.627546 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:48.627618 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:48.711376 (+ 83758us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3109248}
I20260810 18:51:48.712940 16624 rpcz_store.cc:276] 0810 18:51:47.371082 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:47.371198 (+ 116us) service_pool.cc:227] Handling call
0810 18:51:47.430889 (+ 59691us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:51:47.431344 (+ 455us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:47.431384 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:51:47.487056 (+ 55672us) log.cc:842] Serialized 501 byte log entry
0810 18:51:48.234917 (+747861us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:51:48.403829 (+168912us) log.cc:842] Serialized 14 byte log entry
0810 18:51:48.404101 (+ 272us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:48.404196 (+ 95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:51:48.617794 (+213598us) raft_consensus.cc:1671] finished
0810 18:51:48.617821 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:48.618006 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3615,"raft.queue_time_us":111,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":11405,"thread_start_us":734300,"threads_started":1,"wal-append.queue_time_us":1057224}
W20260810 18:51:48.638392 18580 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.722s user 0.033s sys 0.027s
W20260810 18:51:48.509095 16747 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.965s user 0.009s sys 0.034s
W20260810 18:51:48.668304 18709 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.563s user 0.008s sys 0.041s
I20260810 18:51:48.629136 18874 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:51:48.862458 16693 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.222s user 0.081s sys 0.102s
W20260810 18:51:48.862330 16342 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.898s user 0.031s sys 0.029s
I20260810 18:51:48.882277 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:48.870646 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:48.871593 16623 raft_consensus.cc:2454] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 1.
I20260810 18:51:48.522563 18905 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:48.594136 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:48.575652 18502 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:48.898334 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:48.900036 16508 raft_consensus.cc:2454] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:48.555399 18921 raft_consensus.cc:482] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:48.874482 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:51:48.862167 17771 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.251s user 0.023s sys 0.062s
I20260810 18:51:48.827057 18924 raft_consensus.cc:2726] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827) with higher term. Message: Invalid argument: T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3.
I20260810 18:51:48.895155 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:48.904103 16505 raft_consensus.cc:3037] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
W20260810 18:51:48.865586 16621 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.189s user 0.030s sys 0.063s
W20260810 18:51:48.881398 16747 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.366s user 0.020s sys 0.059s
W20260810 18:51:48.875293 18580 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.959s user 0.038s sys 0.037s
I20260810 18:51:48.679220 18852 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:51:48.827986 18941 raft_consensus.cc:482] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:48.903465 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:51:48.884111 18709 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.781s user 0.020s sys 0.041s
I20260810 18:51:48.824290 18733 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:48.554867 18516 raft_consensus.cc:504] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:48.893328 18936 log.cc:924] Time spent T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.265s user 0.000s sys 0.009s
I20260810 18:51:48.909005 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:48.919901 16507 raft_consensus.cc:2454] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:48.865964 18656 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:48.905583 18567 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:48.916939 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:48.925786 16508 raft_consensus.cc:2454] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:48.905503 18921 raft_consensus.cc:504] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:48.923465 16506 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.763s user 0.021s sys 0.016s
I20260810 18:51:48.912963 17778 raft_consensus.cc:2396] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260810 18:51:48.897405 18874 raft_consensus.cc:482] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
W20260810 18:51:48.929844 16506 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.769s user 0.022s sys 0.016s
I20260810 18:51:48.907594 16626 raft_consensus.cc:2454] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 1.
W20260810 18:51:48.936815 16443 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.050s user 0.017s sys 0.044s
I20260810 18:51:48.936947 18827 raft_consensus.cc:482] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:51:48.937225 16443 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.342s user 0.017s sys 0.044s
I20260810 18:51:48.937304 18516 leader_election.cc:288] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:48.937479 18827 raft_consensus.cc:504] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:48.937903 18516 raft_consensus.cc:482] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:48.929026 18567 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:48.940419 18862 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:48.940922 18862 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:48.943048 18827 leader_election.cc:288] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:48.941009 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:48.942842 18862 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:48.952059 18862 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:51:48.952589 18862 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:48.953998 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:48.954564 18862 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:48.937970 18921 leader_election.cc:288] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:48.917743 18872 raft_consensus.cc:482] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:48.900614 18905 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:48.876561 18871 consensus_queue.cc:237] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:48.960026 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:48.955210 18862 raft_consensus.cc:482] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:48.961047 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:48.919970 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:48.938460 18516 raft_consensus.cc:504] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:48.954553 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:48.907377 18502 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:48.971154 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:48.930840 18882 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:48.933339 18538 raft_consensus.cc:482] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:48.930027 18929 raft_consensus.cc:2781] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:51:48.917112 18852 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:48.956509 18883 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:48.955694 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:48.923305 18733 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:48.950976 18926 raft_consensus.cc:482] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:51:48.984838 18662 consensus_meta.cc:287] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.629s user 0.009s sys 0.001s
W20260810 18:51:48.990733 16574 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.565s user 0.005s sys 0.042s
W20260810 18:51:49.051666 16574 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.628s user 0.006s sys 0.042s
I20260810 18:51:49.051561 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:49.052605 16507 raft_consensus.cc:2454] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:48.930905 18656 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:48.980422 18494 raft_consensus.cc:482] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:51:48.983565 18875 raft_consensus.cc:482] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:48.970093 18921 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:48.919904 18941 raft_consensus.cc:504] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:49.006196 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:48.989473 16628 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:51:48.985072 17780 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55570 (request call id 305) took 1934 ms. Trace:
I20260810 18:51:48.972424 18872 raft_consensus.cc:504] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:48.983474 18708 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:51:49.118005 16501 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.210s user 0.044s sys 0.029s
W20260810 18:51:49.123971 16501 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.216s user 0.045s sys 0.029s
I20260810 18:51:48.959249 18874 raft_consensus.cc:504] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:49.121659 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:48.986258 18862 raft_consensus.cc:504] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:49.125248 16507 raft_consensus.cc:2454] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
W20260810 18:51:48.981098 16695 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.340s user 0.010s sys 0.046s
W20260810 18:51:49.132750 16695 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.491s user 0.010s sys 0.046s
W20260810 18:51:49.133421 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 1.49214e+06us
I20260810 18:51:49.001716 18883 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:49.007308 18538 raft_consensus.cc:504] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:49.158516 16504 raft_consensus.cc:2454] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:49.118093 18931 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:49.165216 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:49.055950 18662 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:49.169782 16328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 03cb267b222b440981c94d431e7273bf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 18:51:49.057830 18926 raft_consensus.cc:504] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:49.159504 17726 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.763s user 0.023s sys 0.013s
I20260810 18:51:49.179105 18938 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:49.179368 18502 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:48.487528 16632 rpcz_store.cc:276] 0810 18:51:47.366586 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:47.380485 (+ 13899us) service_pool.cc:227] Handling call
0810 18:51:47.480826 (+100341us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:47.481086 (+ 260us) raft_consensus.cc:1520] Early marking committed up to index 5
0810 18:51:47.481489 (+ 403us) log.cc:842] Serialized 14 byte log entry
0810 18:51:48.361063 (+879574us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:48.364425 (+ 3362us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:51:48.390492 (+ 26067us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:48.412353 (+ 21861us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:48.440817 (+ 28464us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":3017,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":132724,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":5109,"thread_start_us":854732,"threads_started":1}
I20260810 18:51:49.186017 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:51:49.177163 17726 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.787s user 0.023s sys 0.013s
I20260810 18:51:49.158457 18567 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:49.124806 18921 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:49.121199 18494 raft_consensus.cc:504] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:49.121258 18875 raft_consensus.cc:504] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:49.131704 18708 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:49.174592 18906 raft_consensus.cc:482] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:49.238225 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:49.179651 18938 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:49.193457 18852 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:49.272114 17787 raft_consensus.cc:2454] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:49.175840 18931 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:49.271588 18567 raft_consensus.cc:482] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:49.273792 18567 raft_consensus.cc:504] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:49.274385 18921 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:49.274510 18931 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:51:49.275497 18950 log.cc:924] Time spent T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.160s user 0.000s sys 0.001s
I20260810 18:51:49.276067 18567 leader_election.cc:288] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:49.277041 18874 leader_election.cc:288] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:51:49.187402 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 835025us
I20260810 18:51:49.277493 16505 raft_consensus.cc:2454] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
I20260810 18:51:49.177475 18942 raft_consensus.cc:482] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:51:49.293965 16636 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.680s user 0.020s sys 0.050s
W20260810 18:51:49.294330 16636 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.680s user 0.020s sys 0.050s
I20260810 18:51:49.283411 18862 leader_election.cc:288] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:49.282670 18957 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:49.292008 17771 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55586 (request call id 201) took 1681 ms. Trace:
I20260810 18:51:49.307237 16743 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6"
dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
mode: GRACEFUL
new_leader_uuid: "831925dd8b644ebab7eb0d672a94eddd"
from {username='slave'} at 127.0.0.1:45870
I20260810 18:51:49.308146 16743 raft_consensus.cc:595] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Received request to transfer leadership to TS 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:49.289706 18516 leader_election.cc:288] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:49.203884 18709 raft_consensus.cc:482] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:51:49.304733 16573 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.714s user 0.025s sys 0.061s
W20260810 18:51:49.303661 18563 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.829s user 0.023s sys 0.053s
I20260810 18:51:49.240516 18905 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:49.312737 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:49.397789 16750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1266) took 1362 ms. Trace:
I20260810 18:51:49.295728 18827 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
W20260810 18:51:49.315999 16573 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.726s user 0.025s sys 0.062s
I20260810 18:51:49.317080 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:49.302511 18931 raft_consensus.cc:482] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:49.398258 16750 rpcz_store.cc:276] 0810 18:51:47.955220 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:47.955344 (+ 124us) service_pool.cc:227] Handling call
0810 18:51:47.955794 (+ 450us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:47.956040 (+ 246us) raft_consensus.cc:1520] Early marking committed up to index 3
0810 18:51:48.824399 (+868359us) log.cc:842] Serialized 14 byte log entry
0810 18:51:49.316401 (+492002us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:49.316451 (+ 50us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:51:49.316496 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:49.317468 (+ 972us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:49.317603 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1240396,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":2379,"thread_start_us":1251657,"threads_started":2}
I20260810 18:51:49.398658 16507 raft_consensus.cc:2454] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 3.
W20260810 18:51:49.405813 16633 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.629s user 0.002s sys 0.080s
W20260810 18:51:49.406324 16633 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.694s user 0.002s sys 0.080s
I20260810 18:51:49.402279 18905 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:51:49.312443 18957 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:49.408787 18905 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:49.409192 18875 leader_election.cc:288] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:49.311599 18733 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:51:49.397545 18563 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.923s user 0.024s sys 0.053s
I20260810 18:51:49.300807 17726 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:51:49.295678 18942 raft_consensus.cc:504] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:49.419674 16572 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.030s sys 0.049s
I20260810 18:51:49.305397 18906 raft_consensus.cc:504] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:49.118283 17780 rpcz_store.cc:276] 0810 18:51:47.039631 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:47.039757 (+ 126us) service_pool.cc:227] Handling call
0810 18:51:47.085601 (+ 45844us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:51:47.170673 (+ 85072us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:47.179233 (+ 8560us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:51:47.293041 (+113808us) log.cc:842] Serialized 26 byte log entry
0810 18:51:48.305154 (+1012113us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:51:48.305214 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:48.305308 (+ 94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:51:48.955608 (+650300us) raft_consensus.cc:1671] finished
0810 18:51:48.958176 (+ 2568us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:48.971779 (+ 13603us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6797,"raft.queue_time_us":31298,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":19200,"thread_start_us":965173,"threads_started":1,"wal-append.queue_time_us":1193183}
I20260810 18:51:49.413118 16575 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:49.422655 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:51:49.420672 16572 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.597s user 0.030s sys 0.049s
I20260810 18:51:49.403393 18872 leader_election.cc:288] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:49.406558 18921 raft_consensus.cc:482] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:51:49.410279 18937 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:49.415525 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:49.433430 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:51:49.438570 16448 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.007s sys 0.012s
I20260810 18:51:49.411526 18656 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:49.432325 16447 leader_election.cc:302] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:51:49.440392 17724 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.540s user 0.004s sys 0.015s
W20260810 18:51:49.439091 16448 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.528s user 0.007s sys 0.012s
I20260810 18:51:49.438620 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:49.438776 16745 raft_consensus.cc:2454] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:49.422216 18538 leader_election.cc:288] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:49.406180 18931 raft_consensus.cc:504] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:49.443408 16693 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.540s user 0.014s sys 0.061s
W20260810 18:51:49.451713 16693 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.548s user 0.015s sys 0.061s
W20260810 18:51:49.440768 17724 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.541s user 0.004s sys 0.015s
W20260810 18:51:49.445763 16701 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.011s sys 0.009s
I20260810 18:51:49.355804 18709 raft_consensus.cc:504] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:49.449649 16450 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.013s sys 0.007s
W20260810 18:51:49.453544 16450 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.542s user 0.014s sys 0.007s
W20260810 18:51:49.452423 16701 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.554s user 0.011s sys 0.010s
W20260810 18:51:49.457247 16443 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.519s user 0.021s sys 0.002s
I20260810 18:51:49.436303 18733 raft_consensus.cc:2781] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:49.437763 18941 leader_election.cc:288] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:49.407032 18827 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:51:49.458112 16443 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.520s user 0.021s sys 0.002s
I20260810 18:51:49.462430 18942 leader_election.cc:288] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:49.482209 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:49.482707 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:49.482875 16507 raft_consensus.cc:2454] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
W20260810 18:51:49.452276 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 855572us
W20260810 18:51:49.432641 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 134596us
I20260810 18:51:49.441304 18567 raft_consensus.cc:482] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:49.422010 18852 consensus_queue.cc:1035] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20260810 18:51:49.442902 18926 leader_election.cc:288] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:49.487660 18709 leader_election.cc:288] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:51:49.513360 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 199916us
I20260810 18:51:49.511718 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:49.489948 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:49.494338 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:51:49.460353 18580 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.005s sys 0.017s
I20260810 18:51:49.451165 18708 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:49.447885 18516 raft_consensus.cc:482] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:49.459300 18808 raft_consensus.cc:482] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
W20260810 18:51:49.521041 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 108025us
I20260810 18:51:49.534910 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:49.440866 18937 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:49.530339 18580 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.605s user 0.005s sys 0.017s
W20260810 18:51:49.545212 18953 log.cc:924] Time spent T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.273s user 0.001s sys 0.001s
I20260810 18:51:49.544834 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:51:49.493203 18945 log.cc:924] Time spent T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.252s user 0.000s sys 0.001s
I20260810 18:51:49.476037 18494 leader_election.cc:288] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:49.484468 18538 raft_consensus.cc:2781] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:49.558602 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:49.563473 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:49.563612 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:49.563772 16575 leader_election.cc:302] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
W20260810 18:51:49.722510 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 158785us
I20260810 18:51:49.441560 18921 raft_consensus.cc:504] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:49.544485 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:49.499568 18733 raft_consensus.cc:686] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:51:49.512161 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:49.497920 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:49.545781 18957 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:49.544103 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:49.558265 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:49.645671 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:49.747700 18937 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:49.748205 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:49.748914 18937 raft_consensus.cc:482] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:49.749475 18937 raft_consensus.cc:504] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:49.749771 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:49.646070 18963 log.cc:924] Time spent T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.207s user 0.000s sys 0.001s
W20260810 18:51:49.564167 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 48819.7us
I20260810 18:51:49.568243 18808 raft_consensus.cc:504] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:49.640462 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:49.751991 17726 leader_election.cc:302] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:51:49.752357 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:49.753230 16626 raft_consensus.cc:2396] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260810 18:51:49.515403 18567 raft_consensus.cc:504] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:49.653898 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 148651us
I20260810 18:51:49.564114 18516 raft_consensus.cc:504] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:49.656222 18954 log.cc:924] Time spent T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.353s user 0.002s sys 0.000s
I20260810 18:51:49.517016 18874 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:49.557742 18862 raft_consensus.cc:482] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:49.768869 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:49.769150 18862 raft_consensus.cc:504] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:49.769956 18921 leader_election.cc:288] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:49.770534 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:49.638181 18873 consensus_queue.cc:1035] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260810 18:51:49.544332 18948 raft_consensus.cc:482] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:49.773370 16573 leader_election.cc:302] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:49.307523 17771 rpcz_store.cc:276] 0810 18:51:47.609134 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:47.609325 (+ 191us) service_pool.cc:227] Handling call
0810 18:51:47.609784 (+ 459us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:51:47.610203 (+ 419us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:47.610246 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:51:47.610937 (+ 691us) log.cc:842] Serialized 26 byte log entry
0810 18:51:48.913295 (+1302358us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:51:48.915539 (+ 2244us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:48.920232 (+ 4693us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:51:49.277715 (+357483us) raft_consensus.cc:1671] finished
0810 18:51:49.282406 (+ 4691us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:49.290421 (+ 8015us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1574,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":8194,"thread_start_us":1292216,"threads_started":1,"wal-append.queue_time_us":1371882}
I20260810 18:51:49.704524 18964 raft_consensus.cc:482] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:49.753357 18938 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:49.705819 18931 leader_election.cc:288] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:49.707741 18852 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:49.531030 18947 raft_consensus.cc:482] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
W20260810 18:51:49.714567 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 187846us
W20260810 18:51:49.720427 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 226219us
I20260810 18:51:49.739686 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:51:49.720829 18966 log.cc:924] Time spent T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.154s user 0.002s sys 0.002s
I20260810 18:51:49.786727 18947 raft_consensus.cc:504] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:49.787335 16747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43726 (request call id 1219) took 2505 ms. Trace:
I20260810 18:51:49.788720 18567 leader_election.cc:288] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:49.788825 18947 leader_election.cc:288] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:49.789498 18567 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:49.789872 18662 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:49.790362 18516 leader_election.cc:288] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:49.790144 18567 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:49.745985 16496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1238) took 2389 ms. Trace:
I20260810 18:51:49.746395 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:49.779899 18921 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:51:49.813979 16495 raft_consensus.cc:2454] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 1.
I20260810 18:51:49.814064 18921 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:49.815207 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:49.751159 18494 raft_consensus.cc:482] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:49.815898 16495 raft_consensus.cc:2454] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 2.
I20260810 18:51:49.816303 18494 raft_consensus.cc:504] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:49.816980 18516 raft_consensus.cc:979] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be: : Instructing follower 831925dd8b644ebab7eb0d672a94eddd to start an election
I20260810 18:51:49.816886 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:49.817762 16695 leader_election.cc:302] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:49.756934 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:49.723780 18926 raft_consensus.cc:2726] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 18:51:49.732064 18961 log.cc:924] Time spent T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.420s user 0.002s sys 0.003s
I20260810 18:51:49.762398 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:49.752802 18937 leader_election.cc:288] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:49.721148 18538 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:49.845363 18538 raft_consensus.cc:3037] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:49.848762 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:49.849514 16622 raft_consensus.cc:2454] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:49.850076 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:49.850343 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:49.515025 18871 raft_consensus.cc:482] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:49.771701 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:49.851382 18871 raft_consensus.cc:504] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:49.851997 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:49.852638 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:49.853216 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:49.853719 18871 leader_election.cc:288] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:49.854535 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:49.855219 17775 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55602 (request call id 247) took 1442 ms. Trace:
I20260810 18:51:49.855811 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:49.855669 17775 rpcz_store.cc:276] 0810 18:51:48.412611 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:48.426384 (+ 13773us) service_pool.cc:227] Handling call
0810 18:51:49.854424 (+1428040us) spinlock_profiling.cc:243] Waited 1.75 ms on lock 0x7b5800005498. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed0502b0 00007f1aed03b7f9 00007f1aed03b2a2 00007f1ae6721684
0810 18:51:49.854662 (+ 238us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:49.854859 (+ 197us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:49.854888 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:49.854939 (+ 51us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:51:49.854971 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:49.855039 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:49.855164 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4029696}
I20260810 18:51:49.856662 16495 raft_consensus.cc:2454] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:49.774767 16573 leader_election.cc:302] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:49.735148 18906 leader_election.cc:288] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:49.758847 16748 raft_consensus.cc:2454] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:49.741487 16742 raft_consensus.cc:2454] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:49.760432 18905 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:49.780974 18964 raft_consensus.cc:504] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:49.858379 18989 raft_consensus.cc:2781] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:49.858757 18989 raft_consensus.cc:482] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:49.858847 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:49.859161 18989 raft_consensus.cc:3037] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:49.785920 18852 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:49.743132 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:49.783595 18948 raft_consensus.cc:504] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:49.792200 18946 raft_consensus.cc:2781] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:51:49.775617 18874 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:49.861450 16507 raft_consensus.cc:2454] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:49.862461 18874 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:49.792806 16752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1253) took 2469 ms. Trace:
I20260810 18:51:49.792956 18827 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:49.754226 17780 raft_consensus.cc:2454] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:49.863442 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:49.793558 18733 consensus_queue.cc:237] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:49.864091 18883 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:49.864408 18567 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:49.864804 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:49.778107 18970 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:49.794461 16501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1234) took 2452 ms. Trace:
I20260810 18:51:49.783931 18938 raft_consensus.cc:482] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:49.795495 16621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1235) took 2420 ms. Trace:
I20260810 18:51:49.787773 16747 rpcz_store.cc:276] 0810 18:51:47.277969 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:47.278145 (+ 176us) service_pool.cc:227] Handling call
0810 18:51:47.294361 (+ 16216us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:51:47.358045 (+ 63684us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:47.380139 (+ 22094us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:51:47.489608 (+109469us) log.cc:842] Serialized 26 byte log entry
0810 18:51:48.917553 (+1427945us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:51:48.923335 (+ 5782us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:48.929458 (+ 6123us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:51:49.775222 (+845764us) raft_consensus.cc:1671] finished
0810 18:51:49.775249 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:49.783720 (+ 8471us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10664,"raft.queue_time_us":18848,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":44724,"thread_start_us":1410082,"threads_started":1,"wal-append.queue_time_us":1687163}
I20260810 18:51:49.793679 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:49.803500 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:49.805517 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:49.805779 18957 raft_consensus.cc:2726] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:49.807056 19019 raft_consensus.cc:2781] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:49.810473 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:49.865031 18921 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:51:49.824214 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 108487us
I20260810 18:51:49.820580 18931 raft_consensus.cc:2726] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:51:49.824975 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:49.779474 18808 leader_election.cc:288] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:49.825428 18516 raft_consensus.cc:1067] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Signalling peer 831925dd8b644ebab7eb0d672a94eddd to start an election
I20260810 18:51:49.826665 18494 leader_election.cc:288] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:49.828613 16631 raft_consensus.cc:1261] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260810 18:51:49.952174 16631 pending_rounds.cc:76] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb: Aborting all ops after (but not including) 1
I20260810 18:51:49.952476 16631 pending_rounds.cc:97] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260810 18:51:49.952196 16448 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:49.952826 16631 raft_consensus.cc:2866] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:51:49.953614 16631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1239) took 1319 ms. Trace:
I20260810 18:51:49.828891 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:49.830816 16505 raft_consensus.cc:2454] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:49.920047 19019 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:49.797024 18862 leader_election.cc:288] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:49.840943 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:49.953959 16631 rpcz_store.cc:276] 0810 18:51:48.634078 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:49.827996 (+1193918us) service_pool.cc:227] Handling call
0810 18:51:49.828283 (+ 287us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:49.953347 (+125064us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:49.953562 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:51:49.964327 16746 raft_consensus.cc:2454] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:49.841579 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:49.925302 18929 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:49.844048 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:49.844698 16506 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1236) took 2502 ms. Trace:
I20260810 18:51:49.844785 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:49.860472 16636 raft_consensus.cc:1799] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Handling vote request from an unknown peer d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:49.992527 16636 raft_consensus.cc:2396] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260810 18:51:49.909636 18970 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:49.863049 16752 rpcz_store.cc:276] 0810 18:51:47.323913 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:47.324152 (+ 239us) service_pool.cc:227] Handling call
0810 18:51:47.324813 (+ 661us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:51:47.535161 (+210348us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:47.535205 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:51:47.577257 (+ 42052us) log.cc:842] Serialized 501 byte log entry
0810 18:51:48.715282 (+1138025us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:51:48.715343 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:48.940957 (+225614us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b540020b054. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:51:48.941002 (+ 45us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:51:49.792532 (+851530us) raft_consensus.cc:1671] finished
0810 18:51:49.792552 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:49.792751 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":442129280,"thread_start_us":1097918,"threads_started":1,"wal-append.queue_time_us":1645729}
I20260810 18:51:49.871594 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:49.869840 16501 rpcz_store.cc:276] 0810 18:51:47.341861 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:47.344881 (+ 3020us) service_pool.cc:227] Handling call
0810 18:51:47.674503 (+329622us) spinlock_profiling.cc:243] Waited 2.24 ms on lock 0x7b5800001898. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed0502b0 00007f1aed03b7f9 00007f1aed03b2a2 00007f1ae6721684
0810 18:51:47.674743 (+ 240us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:47.797643 (+122900us) raft_consensus.cc:1520] Early marking committed up to index 4
0810 18:51:47.907301 (+109658us) log.cc:842] Serialized 14 byte log entry
0810 18:51:49.124554 (+1217253us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:49.124606 (+ 52us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:51:49.124648 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:49.793448 (+668800us) spinlock_profiling.cc:243] Waited 7.25 ms on lock 0x7b1800262a20. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1aee0718b1 00007f1aee07e661 00007f1ae519ff96 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9
0810 18:51:49.794247 (+ 799us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b54001cbbd4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:51:49.794289 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:49.794404 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":95,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":20523,"spinlock_wait_cycles":303776128,"thread_start_us":1216823,"threads_started":1,"wal-append.queue_time_us":1504931}
I20260810 18:51:49.965822 19026 raft_consensus.cc:2726] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:51:49.995395 16623 raft_consensus.cc:2454] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
W20260810 18:51:49.996122 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 244195us
I20260810 18:51:49.850880 16622 raft_consensus.cc:2454] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:49.812564 16701 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:49.870273 18946 raft_consensus.cc:686] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
W20260810 18:51:49.960780 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 251267us
I20260810 18:51:49.964691 18941 raft_consensus.cc:482] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:51:49.961799 16633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60066 (request call id 74) took 1408 ms. Trace:
I20260810 18:51:49.926882 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:49.811959 16496 rpcz_store.cc:276] 0810 18:51:47.356672 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:47.456039 (+ 99367us) service_pool.cc:227] Handling call
0810 18:51:47.907927 (+451888us) spinlock_profiling.cc:243] Waited 2.57 ms on lock 0x7b5800001898. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed0502b0 00007f1aed03b7f9 00007f1aed03b2a2 00007f1ae6721684
0810 18:51:49.408202 (+1500275us) spinlock_profiling.cc:243] Waited 5.97 ms on lock 0x7b5800001898. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519abce 000000000034aea9 00007f1aecfc6a30 00007f1aed08c7b4 00007f1aed08c821 00007f1aed0502b0 00007f1aed03b7f9 00007f1aed03b2a2 00007f1ae6721684
0810 18:51:49.416182 (+ 7980us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:49.431288 (+ 15106us) raft_consensus.cc:1520] Early marking committed up to index 6
0810 18:51:49.445370 (+ 14082us) log.cc:842] Serialized 14 byte log entry
0810 18:51:49.745555 (+300185us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:49.745625 (+ 70us) raft_consensus.cc:1631] Marking committed up to 6
0810 18:51:49.745683 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:49.745760 (+ 77us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:49.745900 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3380,"prepare.queue_time_us":2172,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":72684,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"spinlock_wait_cycles":19659008,"thread_start_us":249134,"threads_started":1}
I20260810 18:51:49.909977 18938 raft_consensus.cc:504] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:49.966061 19019 raft_consensus.cc:3037] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:49.868319 16624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1240) took 1224 ms. Trace:
I20260810 18:51:49.991523 16506 rpcz_store.cc:276] 0810 18:51:47.342564 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:47.343546 (+ 982us) service_pool.cc:227] Handling call
0810 18:51:47.343883 (+ 337us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:51:47.405101 (+ 61218us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:47.405147 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:51:48.101911 (+696764us) spinlock_profiling.cc:243] Waited 3.17 ms on lock 0x7b2c000307a8. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae6adde11 00007f1ae6aed442 00007f1ae8135c11 00007f1ae80fa58a 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772
0810 18:51:48.102386 (+ 475us) log.cc:842] Serialized 26 byte log entry
0810 18:51:48.936098 (+833712us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:51:48.936151 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:48.936237 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:51:49.844415 (+908178us) raft_consensus.cc:1671] finished
0810 18:51:49.844435 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:49.844641 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":60764,"raft.queue_time_us":60963,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":48818,"spinlock_wait_cycles":7283584,"thread_start_us":775356,"threads_started":1,"wal-append.queue_time_us":1135106}
I20260810 18:51:49.876121 16621 rpcz_store.cc:276] 0810 18:51:47.375108 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:47.595021 (+219913us) service_pool.cc:227] Handling call
0810 18:51:47.603145 (+ 8124us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:51:47.614012 (+ 10867us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:47.620102 (+ 6090us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:51:47.670389 (+ 50287us) log.cc:842] Serialized 26 byte log entry
0810 18:51:48.914624 (+1244235us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:51:48.921222 (+ 6598us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:48.927133 (+ 5911us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:51:49.795221 (+868088us) raft_consensus.cc:1671] finished
0810 18:51:49.795247 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:49.795387 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":43222,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":130,"thread_start_us":1229487,"threads_started":1,"wal-append.queue_time_us":1505195}
I20260810 18:51:49.997006 17726 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
W20260810 18:51:49.917810 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 143106us
I20260810 18:51:49.960875 16448 leader_election.cc:302] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:50.011611 16624 rpcz_store.cc:276] 0810 18:51:48.634454 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:48.869453 (+234999us) service_pool.cc:227] Handling call
0810 18:51:48.869830 (+ 377us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:49.130642 (+260812us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:49.130688 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:49.130745 (+ 57us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:51:49.130786 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:49.858882 (+728096us) spinlock_profiling.cc:243] Waited 261 ms on lock 0x7b54000cdd54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:51:49.858924 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:49.859037 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":599197440}
I20260810 18:51:50.012310 16573 leader_election.cc:302] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:51:50.012686 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 51913.9us
I20260810 18:51:49.999694 16626 raft_consensus.cc:2454] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:49.999627 16450 leader_election.cc:302] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:51:50.000293 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 187808us
I20260810 18:51:50.014533 16448 leader_election.cc:302] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:49.998303 18946 consensus_queue.cc:237] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:50.008734 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 244144us
I20260810 18:51:50.009234 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:50.016911 16508 raft_consensus.cc:2454] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:50.017473 16508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52338 (request call id 1204) took 1081 ms. Trace:
I20260810 18:51:49.973344 18929 raft_consensus.cc:3037] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:50.001979 18970 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:50.016327 18948 leader_election.cc:288] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:51:50.104768 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 205842us
I20260810 18:51:50.017813 16508 rpcz_store.cc:276] 0810 18:51:48.936359 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:48.936503 (+ 144us) service_pool.cc:227] Handling call
0810 18:51:50.009081 (+1072578us) spinlock_profiling.cc:243] Waited 19.1 ms on lock 0x7b5800001898. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed03c1b0 00007f1aed03ba94 00007f1ae6723c04 00007f1ae6723b92
0810 18:51:50.017423 (+ 8342us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":43946368}
W20260810 18:51:50.098205 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 76347.3us
I20260810 18:51:50.098779 17787 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6"
dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
from {username='slave'} at 127.0.0.1:55586
W20260810 18:51:50.105394 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 88634.9us
I20260810 18:51:50.021710 16573 leader_election.cc:302] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:50.106142 17724 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:49.994261 17778 raft_consensus.cc:2454] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:50.014137 18809 raft_consensus.cc:2781] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:51:50.108314 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:50.002767 16633 rpcz_store.cc:276] 0810 18:51:48.553957 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:48.554380 (+ 423us) service_pool.cc:227] Handling call
0810 18:51:48.554862 (+ 482us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:51:49.406753 (+851891us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:49.406789 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:51:49.407265 (+ 476us) log.cc:842] Serialized 26 byte log entry
0810 18:51:49.554624 (+147359us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:51:49.959217 (+404593us) log.cc:842] Serialized 14 byte log entry
0810 18:51:49.961487 (+ 2270us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:49.961574 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:51:49.961616 (+ 42us) raft_consensus.cc:1671] finished
0810 18:51:49.961627 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:49.961747 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":199133,"prepare.queue_time_us":164978,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":71,"raft.queue_time_us":403544,"raft.run_cpu_time_us":296,"raft.run_wall_time_us":11061,"thread_start_us":1043561,"threads_started":3,"wal-append.queue_time_us":369149}
I20260810 18:51:50.118351 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:50.108513 17724 leader_election.cc:302] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:50.108592 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:50.107239 16448 leader_election.cc:302] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:49.992668 17771 raft_consensus.cc:2454] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
W20260810 18:51:50.111032 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 89426.9us
I20260810 18:51:50.126094 16573 leader_election.cc:302] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:50.127323 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:50.127761 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:50.111500 16450 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:50.016216 19026 raft_consensus.cc:2726] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:50.019069 19024 raft_consensus.cc:2726] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:50.122327 16448 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:50.015482 18964 leader_election.cc:288] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:50.021014 18957 raft_consensus.cc:2781] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:50.123764 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:50.104038 18905 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:50.103792 18874 raft_consensus.cc:482] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:50.133490 16695 leader_election.cc:302] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:50.133688 16573 leader_election.cc:302] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:50.132818 16450 leader_election.cc:302] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:50.015765 18941 raft_consensus.cc:504] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:50.134732 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:50.143123 16695 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:50.136162 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:50.134264 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:50.117282 18809 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:50.126368 18982 raft_consensus.cc:2781] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:50.126644 18862 raft_consensus.cc:2726] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:51:50.142077 16450 leader_election.cc:302] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:50.147967 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:50.131893 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:50.153251 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:50.154081 16573 leader_election.cc:302] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:50.132449 18852 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:51:50.131667 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 134770us
W20260810 18:51:50.189239 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 47239.6us
I20260810 18:51:50.343652 16624 raft_consensus.cc:2454] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:51:50.183774 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:50.344202 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 235742us
I20260810 18:51:50.162171 16504 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52338 (request call id 1203) took 1233 ms. Trace:
I20260810 18:51:50.156354 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:50.345199 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 191228us
I20260810 18:51:50.138372 18883 raft_consensus.cc:2726] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:51:50.134500 18730 raft_consensus.cc:2781] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:50.384862 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:50.391973 16504 rpcz_store.cc:276] 0810 18:51:48.928460 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:48.931450 (+ 2990us) service_pool.cc:227] Handling call
0810 18:51:50.161984 (+1230534us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b54000f5854. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:50.162102 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":434647424}
I20260810 18:51:50.143525 18964 raft_consensus.cc:2726] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:50.134611 18957 raft_consensus.cc:482] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:50.393569 17724 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:50.383183 18937 raft_consensus.cc:2781] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:50.143251 18859 raft_consensus.cc:2781] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:50.398486 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:50.421797 16508 raft_consensus.cc:2454] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
I20260810 18:51:50.183327 18922 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:51:50.397563 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:50.146538 19026 raft_consensus.cc:2781] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:50.214116 18580 raft_consensus.cc:1050] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea: attempting to promote NON_VOTER 831925dd8b644ebab7eb0d672a94eddd to VOTER
I20260810 18:51:50.394618 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:50.150197 18809 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:50.148000 18631 raft_consensus.cc:2781] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:50.395232 18730 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:50.146461 18921 raft_consensus.cc:2781] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:50.423619 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:50.469527 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:50.469821 16504 raft_consensus.cc:1799] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Handling vote request from an unknown peer 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:50.470568 16504 raft_consensus.cc:2396] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260810 18:51:50.277891 18926 raft_consensus.cc:482] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:50.472193 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:51:50.161036 19023 log.cc:924] Time spent T 00000000000000000000000000000000 P 0975c6fa18274f2baee7fb3615d5b985: Append to log took a long time: real 0.144s user 0.002s sys 0.001s
I20260810 18:51:50.139580 18970 raft_consensus.cc:2781] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:50.212877 18870 raft_consensus.cc:2781] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:50.425652 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:50.465319 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:50.187366 18827 raft_consensus.cc:2781] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:50.474749 16504 raft_consensus.cc:2454] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:50.140368 18905 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:50.385515 18852 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:51:50.477696 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:50.395349 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:50.423532 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:50.160856 19017 raft_consensus.cc:2781] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:50.184031 19018 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:50.419739 18957 raft_consensus.cc:3037] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:50.492935 16746 raft_consensus.cc:2454] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:50.186961 18982 raft_consensus.cc:482] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:50.183552 19029 raft_consensus.cc:2726] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:50.152364 18874 raft_consensus.cc:504] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:50.469561 18730 raft_consensus.cc:3037] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:50.470610 18938 leader_election.cc:288] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:50.467700 18631 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:50.506317 16575 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:50.469725 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:50.469594 18921 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:50.430594 18859 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:51:50.512210 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 118737us
I20260810 18:51:50.512311 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:50.560619 17724 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:50.316713 18808 raft_consensus.cc:482] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:50.513221 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:50.517076 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:50.562160 16508 raft_consensus.cc:2379] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 3: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:50.420058 18937 raft_consensus.cc:482] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
W20260810 18:51:50.470280 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 343148us
I20260810 18:51:50.478271 18870 raft_consensus.cc:482] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:50.487581 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:50.482949 18970 raft_consensus.cc:482] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:50.488637 18827 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:50.469745 18922 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:50.466930 19026 raft_consensus.cc:482] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
W20260810 18:51:50.491353 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 60337.5us
I20260810 18:51:50.536211 18982 raft_consensus.cc:3037] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:50.538141 19017 raft_consensus.cc:482] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:50.562346 18859 raft_consensus.cc:3037] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:50.559024 18921 raft_consensus.cc:3037] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:50.542105 19038 raft_consensus.cc:2781] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:51:50.537901 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:50.499877 18926 raft_consensus.cc:504] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:50.554447 18631 raft_consensus.cc:3037] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:50.568136 17724 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:51:50.568254 19033 log.cc:924] Time spent T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.348s user 0.001s sys 0.005s
I20260810 18:51:50.552021 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:50.569651 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:50.510394 18852 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:50.584329 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:50.603806 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:50.566524 18870 raft_consensus.cc:3037] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:50.564332 18937 raft_consensus.cc:3037] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:50.579218 16740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1283) took 1122 ms. Trace:
I20260810 18:51:50.569947 18827 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:50.575816 19029 raft_consensus.cc:2781] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:50.618357 19029 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:50.618764 19029 raft_consensus.cc:3037] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:50.578496 19017 raft_consensus.cc:3037] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:50.577198 18874 leader_election.cc:288] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:50.608654 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:50.620944 16508 raft_consensus.cc:2379] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 3: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:50.580351 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:50.609458 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:50.610298 16506 raft_consensus.cc:2454] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:50.554453 19018 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:50.575829 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:50.604816 16742 raft_consensus.cc:2454] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:50.630162 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:50.631431 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:50.631899 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:50.633338 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:50.634052 18942 raft_consensus.cc:482] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:50.634316 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:50.616477 16740 rpcz_store.cc:276] 0810 18:51:49.456370 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:49.459017 (+ 2647us) service_pool.cc:227] Handling call
0810 18:51:50.504644 (+1045627us) spinlock_profiling.cc:243] Waited 6.72 ms on lock 0x7b5800270f98. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519abce 000000000034aea9 00007f1aecfc6a30 00007f1aed08c7b4 00007f1aed08c821 00007f1aed0502b0 00007f1aed03b7f9 00007f1aed03b2a2 00007f1ae6721684
0810 18:51:50.536001 (+ 31357us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:50.554185 (+ 18184us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:50.557388 (+ 3203us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:50.559156 (+ 1768us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:51:50.563257 (+ 4101us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:50.567944 (+ 4687us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:50.579151 (+ 11207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15459712}
I20260810 18:51:50.636026 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:50.637888 16750 raft_consensus.cc:1261] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:51:50.644170 18709 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:50.644902 18709 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:50.615352 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:50.575289 18970 raft_consensus.cc:3037] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:50.617533 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:50.634581 18942 raft_consensus.cc:504] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:50.563959 18808 raft_consensus.cc:504] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:51:50.575974 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:50.624120 16501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1250) took 1107 ms. Trace:
I20260810 18:51:50.667798 16501 rpcz_store.cc:276] 0810 18:51:49.516879 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:49.995583 (+478704us) service_pool.cc:227] Handling call
0810 18:51:49.995913 (+ 330us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:51:49.996247 (+ 334us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:49.996279 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:51:49.996801 (+ 522us) log.cc:842] Serialized 26 byte log entry
0810 18:51:50.135176 (+138375us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:51:50.135299 (+ 123us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:50.135381 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:51:50.623918 (+488537us) raft_consensus.cc:1671] finished
0810 18:51:50.623940 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:50.624063 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3697,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":102,"thread_start_us":136923,"threads_started":1,"wal-append.queue_time_us":543518}
I20260810 18:51:50.669486 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:50.588229 19038 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:50.627707 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:50.692005 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:50.655419 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:50.661476 18874 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:50.661960 18728 raft_consensus.cc:482] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:50.662673 16507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52338 (request call id 1215) took 1166 ms. Trace:
I20260810 18:51:50.664288 18656 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:50.664713 19039 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:50.277102 19024 raft_consensus.cc:2932] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
W20260810 18:51:50.694315 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 557780us
I20260810 18:51:50.695330 16507 rpcz_store.cc:276] 0810 18:51:49.496131 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:49.563385 (+ 67254us) service_pool.cc:227] Handling call
0810 18:51:50.662502 (+1099117us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b5400333954. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:50.662620 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":320191232}
I20260810 18:51:50.711244 16448 leader_election.cc:302] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:50.575558 19026 raft_consensus.cc:3037] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:50.712767 16448 leader_election.cc:302] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:50.715337 16448 leader_election.cc:302] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:50.716727 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:50.726579 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:51:50.744596 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 196045us
I20260810 18:51:50.695349 19038 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:50.716485 18883 raft_consensus.cc:2781] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:50.717185 18872 raft_consensus.cc:2726] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 18:51:50.764964 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 258775us
I20260810 18:51:50.765727 16695 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:50.766656 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:50.766719 16575 leader_election.cc:302] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:50.771142 19047 raft_consensus.cc:2726] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:50.771397 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:50.710927 18728 raft_consensus.cc:504] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:50.711130 18874 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:50.714126 18656 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:50.760169 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:50.764132 18883 raft_consensus.cc:482] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:50.764398 18852 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:50.799487 16508 raft_consensus.cc:2454] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:50.759768 18567 consensus_queue.cc:1035] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260810 18:51:50.779522 18905 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:50.782694 18949 raft_consensus.cc:2726] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:50.800828 18808 leader_election.cc:288] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:50.728287 19039 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:50.822371 19039 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:51:50.824937 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 109685us
I20260810 18:51:50.826046 16448 leader_election.cc:302] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:50.792274 18872 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:50.799571 18883 raft_consensus.cc:3037] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:50.803416 18938 raft_consensus.cc:2726] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:51:50.801753 18922 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:51:50.844841 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 54336.7us
I20260810 18:51:50.803907 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:50.807562 18942 leader_election.cc:288] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:50.789410 19026 raft_consensus.cc:504] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:50.824071 16621 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa"
dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
mode: GRACEFUL
new_leader_uuid: "d304ac47fdf846de81337057a4ef7bea"
from {username='slave'} at 127.0.0.1:59734
I20260810 18:51:50.833676 16632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1237) took 1344 ms. Trace:
I20260810 18:51:50.844622 18926 leader_election.cc:288] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:51:50.880640 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 53151.3us
I20260810 18:51:50.864228 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:51:50.875054 19019 consensus_meta.cc:287] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.777s user 0.020s sys 0.000s
W20260810 18:51:50.895340 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 94781.9us
I20260810 18:51:50.844802 18982 raft_consensus.cc:504] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:50.889145 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 123532us
W20260810 18:51:50.891784 18538 consensus_meta.cc:287] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 1.046s user 0.014s sys 0.000s
I20260810 18:51:50.882751 16621 raft_consensus.cc:595] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Received request to transfer leadership to TS d304ac47fdf846de81337057a4ef7bea
W20260810 18:51:50.891949 18989 consensus_meta.cc:287] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 1.032s user 0.003s sys 0.011s
I20260810 18:51:50.902326 16573 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:50.864975 18872 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:51:50.892418 18929 consensus_meta.cc:287] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.789s user 0.003s sys 0.012s
I20260810 18:51:50.890761 18730 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:50.890120 18949 raft_consensus.cc:2726] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260810 18:51:50.911088 19047 raft_consensus.cc:2781] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:50.913895 19047 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:50.914456 19047 raft_consensus.cc:3037] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:50.913795 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:50.916129 16695 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:50.893961 18957 raft_consensus.cc:504] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:50.920074 18567 raft_consensus.cc:2781] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:50.920486 18567 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:50.920830 18567 raft_consensus.cc:3037] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:50.921214 18957 leader_election.cc:288] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:50.914866 18730 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:50.915628 16504 raft_consensus.cc:2379] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 2: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:50.897771 16742 raft_consensus.cc:2454] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 1.
I20260810 18:51:50.924540 17727 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:50.925490 19039 raft_consensus.cc:2781] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:50.925938 19039 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:50.926326 19039 raft_consensus.cc:3037] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:50.919277 16695 leader_election.cc:302] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:50.925535 16505 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1247) took 1409 ms. Trace:
I20260810 18:51:50.931200 16505 rpcz_store.cc:276] 0810 18:51:49.516089 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:49.960581 (+444492us) service_pool.cc:227] Handling call
0810 18:51:50.662885 (+702304us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:50.663049 (+ 164us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:50.663104 (+ 55us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:50.663148 (+ 44us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:51:50.663185 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:50.663239 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:50.925354 (+262115us) spinlock_profiling.cc:243] Waited 702 ms on lock 0x7b5400333950. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:50.925461 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1614487424}
W20260810 18:51:50.934263 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 72700.7us
I20260810 18:51:50.935731 18942 raft_consensus.cc:482] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:50.936420 18942 raft_consensus.cc:504] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:50.910163 18538 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:50.940320 18982 leader_election.cc:288] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:50.941170 18538 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:50.942569 16575 leader_election.cc:302] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:50.941804 19053 raft_consensus.cc:482] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:50.945951 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:50.947412 16505 raft_consensus.cc:2454] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:50.949920 16573 leader_election.cc:302] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:51:50.988505 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 106178us
I20260810 18:51:50.918270 18709 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:50.927161 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:50.941800 19050 raft_consensus.cc:2781] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:50.885916 18580 consensus_queue.cc:237] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:50.895118 19019 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:50.942224 18942 leader_election.cc:288] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:50.951717 18957 raft_consensus.cc:2726] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:50.992914 18728 leader_election.cc:288] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:51.024569 18941 leader_election.cc:288] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:51.025552 19024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:52330 (request call id 787) took 5274 ms. Trace:
I20260810 18:51:51.047305 19076 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:51.032236 18709 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:51.047219 19024 rpcz_store.cc:276] 0810 18:51:45.742326 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:45.751358 (+ 9032us) service_pool.cc:227] Handling call
0810 18:51:45.792472 (+ 41114us) spinlock_profiling.cc:243] Waited 4.9 ms on lock 0x7b5800001898. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed0505b0 00007f1aed03cbe9 00007f1aed03c652 00007f1ae6726184
0810 18:51:46.105649 (+313177us) log.cc:842] Serialized 501 byte log entry
0810 18:51:46.633117 (+527468us) spinlock_profiling.cc:243] Waited 1.25 ms on lock 0x7b54001db620. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae808ba4f 00007f1ae8074545 00007f1ae8079771 00007f1ae80796f9 00007f1ae8079689 00007f1ae8079651 00007f1ae807881d 00007f1aee436204 00007f1aee436039 00007f1ae51d26b6
0810 18:51:46.795003 (+161886us) spinlock_profiling.cc:243] Waited 3.65 ms on lock 0x7b54001db620. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae8093416 00007f1ae808cdaa 00007f1ae808c014 00007f1ae8074545 00007f1ae8079771 00007f1ae80796f9 00007f1ae8079689 00007f1ae8079651 00007f1ae807881d 00007f1aee436204
0810 18:51:51.016338 (+4221335us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7885,"raft.queue_time_us":20253,"raft.run_cpu_time_us":5949,"raft.run_wall_time_us":655231,"spinlock_wait_cycles":22519168,"thread_start_us":312950,"threads_started":1,"wal-append.queue_time_us":379225,"wal-append.run_cpu_time_us":1831,"wal-append.run_wall_time_us":1015642}
I20260810 18:51:51.048146 19076 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:51.048629 18709 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:51.010771 18922 raft_consensus.cc:2781] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:51.031385 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:51.011835 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:50.911960 18989 raft_consensus.cc:504] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:50.912645 18929 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:51.053004 17787 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:51:51.058174 17787 raft_consensus.cc:3037] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:51.054893 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:51.052325 18922 raft_consensus.cc:482] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:51.061751 18922 raft_consensus.cc:3037] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:51.000937 18656 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:51.004487 19050 raft_consensus.cc:482] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:51.067617 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:51:51.071193 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 128754us
I20260810 18:51:51.074564 16575 leader_election.cc:302] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
W20260810 18:51:51.071630 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 121840us
I20260810 18:51:51.075511 16573 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:51.076783 18938 raft_consensus.cc:2781] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:51.077179 18938 raft_consensus.cc:482] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:51.077533 18938 raft_consensus.cc:3037] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:51.079532 16623 raft_consensus.cc:1226] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
W20260810 18:51:51.079627 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 86506.3us
I20260810 18:51:51.080420 16448 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:51.053598 16500 raft_consensus.cc:1226] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:51.050902 18874 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:51.082332 16573 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:51.082927 16575 leader_election.cc:302] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:51.084646 16448 leader_election.cc:302] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:51.096227 16448 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:51.101493 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:51.110826 16575 leader_election.cc:302] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:51.070948 18929 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:50.996294 19053 raft_consensus.cc:504] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:51.112954 16573 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:51.114740 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:51.082295 19050 raft_consensus.cc:3037] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:51.129637 16573 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:51.114799 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:51.120689 16575 leader_election.cc:302] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:51.133239 16621 raft_consensus.cc:1261] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:51:51.133353 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:51.135113 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:51.135485 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:51.136940 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:51.138361 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:51.138691 18631 consensus_meta.cc:287] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.537s user 0.010s sys 0.008s
I20260810 18:51:51.139330 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:51.135136 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:51.086987 18942 raft_consensus.cc:2781] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:51:51.086046 19073 raft_consensus.cc:2726] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:51.144431 16505 raft_consensus.cc:1226] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:51.138990 16633 raft_consensus.cc:2454] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:51.088766 18957 raft_consensus.cc:2726] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 18:51:51.149394 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 151944us
I20260810 18:51:51.093678 18941 raft_consensus.cc:2726] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:51:51.151047 16575 leader_election.cc:302] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
W20260810 18:51:51.138681 19070 log.cc:924] Time spent T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.106s user 0.000s sys 0.000s
I20260810 18:51:51.151157 16573 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:51.115624 18874 raft_consensus.cc:2781] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:51.119814 18728 raft_consensus.cc:2781] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:51.140555 19071 raft_consensus.cc:2726] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:51.167457 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:51.173504 16628 raft_consensus.cc:2454] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:51.174599 16695 leader_election.cc:302] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:51.175323 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:51.168507 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:51.168530 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:51.169123 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:51.132607 19026 leader_election.cc:288] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:51.137071 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:51.127961 19024 raft_consensus.cc:482] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:51.148072 18929 raft_consensus.cc:2726] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:51:51.145653 18942 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:51.203715 16448 leader_election.cc:302] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:51.183689 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:51.205235 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 54288.1us
I20260810 18:51:51.147418 16500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1211) took 1665 ms. Trace:
I20260810 18:51:51.152892 19013 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:51.152251 18905 raft_consensus.cc:3037] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:51.157249 19066 raft_consensus.cc:482] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:51.185565 16323 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet cdfc652dd163415b94d3f0bc3931a07f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 18:51:51.170176 18728 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:51:51.217172 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:51.191183 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:51.207598 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:51.207968 16633 raft_consensus.cc:2379] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 4: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:51.219007 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:51:51.219300 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 68256.3us
I20260810 18:51:51.157446 19052 raft_consensus.cc:2726] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:51:51.212527 16448 leader_election.cc:302] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:51.220378 16573 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:51.173496 18957 raft_consensus.cc:2726] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:51:51.164047 18874 raft_consensus.cc:482] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:51.231418 17787 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:51.218140 17786 raft_consensus.cc:3037] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Advancing to term 2
W20260810 18:51:51.212561 16323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 162714us
I20260810 18:51:51.151705 18631 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:51:51.242409 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 67908.5us
I20260810 18:51:50.878818 16632 rpcz_store.cc:276] 0810 18:51:49.489283 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:49.972388 (+483105us) service_pool.cc:227] Handling call
0810 18:51:50.104874 (+132486us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:50.124326 (+ 19452us) raft_consensus.cc:1520] Early marking committed up to index 3
0810 18:51:50.147854 (+ 23528us) log.cc:842] Serialized 14 byte log entry
0810 18:51:50.645659 (+497805us) spinlock_profiling.cc:243] Waited 26.1 ms on lock 0x7b180028fd80. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1aee0718b1 00007f1aee07e661 00007f1ae51b4345 00007f1aedfeba39 00007f1ae51d18bb 00007f1ae51cfcbf 00007f1ae51d1e22 00007f1ae7d7b5bb 00007f1ae7d81063 00007f1ae7d8157b 00007f1ae810488a
0810 18:51:50.787389 (+141730us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:50.793030 (+ 5641us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:51:50.796672 (+ 3642us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:50.802226 (+ 5554us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:50.823914 (+ 21688us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1796,"prepare.queue_time_us":20337,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":92202,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":8281,"spinlock_wait_cycles":60053760,"thread_start_us":123300,"threads_started":1,"wal-append.queue_time_us":690082}
I20260810 18:51:51.200663 18941 raft_consensus.cc:2781] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:51.248802 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:51.209152 18942 raft_consensus.cc:3037] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:51.196913 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:51.202594 18730 consensus_queue.cc:1035] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20260810 18:51:51.225216 18728 raft_consensus.cc:3037] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:51.216617 18862 raft_consensus.cc:2781] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:51.258873 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:51.248342 19068 raft_consensus.cc:482] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:51.261586 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:51.202854 16621 raft_consensus.cc:2454] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:51.209448 18872 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:51.242679 18874 raft_consensus.cc:3037] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:51.267556 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:51.240742 18949 raft_consensus.cc:2781] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:51.274147 16745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43726 (request call id 1242) took 1163 ms. Trace:
I20260810 18:51:51.247397 18808 raft_consensus.cc:2781] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:51.253968 18957 raft_consensus.cc:2781] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:51.280290 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:51.256191 18941 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:51.217511 19024 raft_consensus.cc:504] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:51.278774 18937 consensus_meta.cc:287] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.659s user 0.018s sys 0.017s
I20260810 18:51:51.266223 19019 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:51.291706 16448 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:51.288966 16745 rpcz_store.cc:276] 0810 18:51:50.110225 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:50.110322 (+ 97us) service_pool.cc:227] Handling call
0810 18:51:51.272385 (+1162063us) spinlock_profiling.cc:243] Waited 3.09 ms on lock 0x7b5800270f98. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed0502b0 00007f1aed03b7f9 00007f1aed03b2a2 00007f1ae6721684
0810 18:51:51.273483 (+ 1098us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:51.273713 (+ 230us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:51.273753 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:51.273809 (+ 56us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:51:51.273873 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:51.273945 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:51.274073 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7105280}
I20260810 18:51:51.276707 16740 raft_consensus.cc:1226] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:51.207108 16500 rpcz_store.cc:276] 0810 18:51:49.482116 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:49.491401 (+ 9285us) service_pool.cc:227] Handling call
0810 18:51:50.997443 (+1506042us) spinlock_profiling.cc:243] Waited 20.7 ms on lock 0x7b5800001898. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519abce 000000000034aea9 00007f1aecfc6a30 00007f1aed08c7b4 00007f1aed08c821 00007f1aed0502b0 00007f1aed03b7f9 00007f1aed03b2a2 00007f1ae6721684
0810 18:51:51.007465 (+ 10022us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:51.089220 (+ 81755us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:51.140190 (+ 50970us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":47612800}
I20260810 18:51:51.290840 18808 raft_consensus.cc:482] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
W20260810 18:51:51.310549 19017 consensus_meta.cc:287] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.686s user 0.012s sys 0.036s
I20260810 18:51:51.305181 17786 raft_consensus.cc:2454] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:51.324796 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:51.326283 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:51.309600 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:51.312234 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 98282.5us
W20260810 18:51:51.283875 18809 consensus_meta.cc:287] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.816s user 0.028s sys 0.022s
I20260810 18:51:51.274658 18862 raft_consensus.cc:482] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:51.328909 18862 raft_consensus.cc:3037] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:51.328588 18809 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:51.293908 19076 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:51:51.290865 18870 consensus_meta.cc:287] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.666s user 0.009s sys 0.013s
I20260810 18:51:51.225443 19066 raft_consensus.cc:504] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:51.290891 18989 leader_election.cc:288] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:51.355924 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:51.322425 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:51.304900 18957 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:51.322366 18808 raft_consensus.cc:3037] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:51.288872 18949 raft_consensus.cc:482] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:51.330920 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:51.307935 18941 raft_consensus.cc:3037] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:51.300027 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:51.224306 19013 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:51.349713 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:51.310223 18709 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:51:51.320304 18859 consensus_meta.cc:287] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.739s user 0.017s sys 0.009s
I20260810 18:51:51.318960 19073 raft_consensus.cc:2781] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:51.360894 16747 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "649d43610ba24875a4418e39b25126ba"
dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
mode: GRACEFUL
new_leader_uuid: "831925dd8b644ebab7eb0d672a94eddd"
from {username='slave'} at 127.0.0.1:45870
I20260810 18:51:51.398974 16747 raft_consensus.cc:595] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Received request to transfer leadership to TS 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:51.366180 16636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1256) took 1247 ms. Trace:
W20260810 18:51:51.349959 19038 consensus_meta.cc:287] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.571s user 0.012s sys 0.000s
I20260810 18:51:51.399730 16636 rpcz_store.cc:276] 0810 18:51:50.109502 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:50.109635 (+ 133us) service_pool.cc:227] Handling call
0810 18:51:51.355760 (+1246125us) spinlock_profiling.cc:243] Waited 1.19 ms on lock 0x7b5800004298. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed03c1b0 00007f1aed03ba94 00007f1ae6723c04 00007f1ae6723b92
0810 18:51:51.356808 (+ 1048us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2738048}
W20260810 18:51:51.320324 19029 consensus_meta.cc:287] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.701s user 0.012s sys 0.012s
I20260810 18:51:51.336220 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:51.315721 18937 raft_consensus.cc:504] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:51.366559 18957 raft_consensus.cc:3037] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:51.373705 18949 raft_consensus.cc:3037] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:51.352602 18871 raft_consensus.cc:482] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:51:51.401682 16749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43726 (request call id 1243) took 1275 ms (client timeout 1595 ms). Trace:
I20260810 18:51:51.279728 19068 raft_consensus.cc:504] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:51.401700 19092 raft_consensus.cc:482] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:51:51.418629 18827 consensus_meta.cc:287] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.800s user 0.011s sys 0.005s
W20260810 18:51:51.423524 16749 rpcz_store.cc:269] 0810 18:51:50.110514 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:50.110650 (+ 136us) service_pool.cc:227] Handling call
0810 18:51:51.349537 (+1238887us) spinlock_profiling.cc:243] Waited 6.66 ms on lock 0x7b5800270f98. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed03c1b0 00007f1aed03ba94 00007f1ae6723c04 00007f1ae6723b92
0810 18:51:51.385590 (+ 36053us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15320320}
I20260810 18:51:51.398581 19073 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
W20260810 18:51:51.418632 18970 consensus_meta.cc:287] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.770s user 0.010s sys 0.010s
I20260810 18:51:51.355670 18870 raft_consensus.cc:504] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:51.403220 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:51.365448 18922 raft_consensus.cc:504] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:51.442726 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:51.444687 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:51.427237 17783 raft_consensus.cc:1226] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 3. Ops: []
I20260810 18:51:51.449247 16701 leader_election.cc:302] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:51.331305 19017 raft_consensus.cc:504] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:51.452656 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:51.453821 16701 leader_election.cc:302] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:51.454674 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:51.457161 16626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 1234) took 1970 ms. Trace:
I20260810 18:51:51.460464 16701 leader_election.cc:302] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:51.407861 19038 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:51.462960 16701 leader_election.cc:302] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:51.463514 19038 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:51.414180 19029 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:51.466202 19029 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:51.458474 16745 raft_consensus.cc:2454] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
I20260810 18:51:51.467520 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:51.433270 18631 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:51.444468 19073 raft_consensus.cc:3037] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:51.470734 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:51.407286 18859 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:51.458636 16626 rpcz_store.cc:276] 0810 18:51:49.487082 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:49.854286 (+367204us) service_pool.cc:227] Handling call
0810 18:51:51.457035 (+1602749us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b54002a4754. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:51.457113 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":331999104}
I20260810 18:51:51.479645 17780 raft_consensus.cc:1226] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:51.449856 19053 leader_election.cc:288] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:51.452394 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:51.436120 18871 raft_consensus.cc:504] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:51.469050 18494 raft_consensus.cc:2726] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260810 18:51:51.500711 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 51315.3us
I20260810 18:51:51.452458 16500 raft_consensus.cc:2454] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:51.442497 18827 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:51.445168 18970 raft_consensus.cc:504] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:51.488713 19024 leader_election.cc:288] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:51.485792 18631 raft_consensus.cc:2726] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:51:51.436563 19092 raft_consensus.cc:504] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:51.510738 19099 raft_consensus.cc:2726] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 18:51:51.497681 18567 consensus_meta.cc:287] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.576s user 0.011s sys 0.019s
I20260810 18:51:51.509059 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:51.554360 16628 raft_consensus.cc:2454] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:51.516305 19013 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:51.497318 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:51.501577 16495 raft_consensus.cc:2379] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 3: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
W20260810 18:51:51.501525 18921 consensus_meta.cc:287] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.896s user 0.010s sys 0.007s
I20260810 18:51:51.550323 16508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52352 (request call id 1266) took 1090 ms. Trace:
I20260810 18:51:51.568390 16508 rpcz_store.cc:276] 0810 18:51:50.460047 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:50.655100 (+195053us) service_pool.cc:227] Handling call
0810 18:51:51.550110 (+895010us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x7b54000f6254. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:51.550258 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":244224384}
I20260810 18:51:51.508561 17775 raft_consensus.cc:2379] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 4: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:51:51.569948 16743 raft_consensus.cc:1799] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Handling vote request from an unknown peer 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:51.507525 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:51.508186 19068 leader_election.cc:288] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:51.666008 17787 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:51.719182 16743 raft_consensus.cc:2379] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 2: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:51:51.671860 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:51.656060 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:51.671443 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:51.720747 16633 raft_consensus.cc:2454] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 1.
I20260810 18:51:51.669283 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:51.653349 16505 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1226) took 1182 ms. Trace:
I20260810 18:51:51.722321 16505 rpcz_store.cc:276] 0810 18:51:50.471199 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:50.949333 (+478134us) service_pool.cc:227] Handling call
0810 18:51:51.144242 (+194909us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:51.153441 (+ 9199us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:51.653105 (+499664us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b54001cb6d0. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:51.653280 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":442009344}
I20260810 18:51:51.723356 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
W20260810 18:51:51.724428 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 58648us
I20260810 18:51:51.609045 16508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1227) took 1109 ms. Trace:
I20260810 18:51:51.728082 16508 rpcz_store.cc:276] 0810 18:51:50.499295 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:51.607939 (+1108644us) service_pool.cc:227] Handling call
0810 18:51:51.608418 (+ 479us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:51.608632 (+ 214us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:51.608668 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:51.608719 (+ 51us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:51:51.608759 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:51.608826 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:51.608965 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:51:51.729964 17787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:55586 (request call id 216) took 1710 ms. Trace:
I20260810 18:51:51.730330 17787 rpcz_store.cc:276] 0810 18:51:50.009201 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:50.009321 (+ 120us) service_pool.cc:227] Handling call
0810 18:51:51.549002 (+1539681us) spinlock_profiling.cc:243] Waited 947 ms on lock 0x7b54005401d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f1da7 00007f1aed03dee0 00007f1ae672f784 00007f1ae672f712 00007f1ae672f641 00007f1ae672f5bc 00007f1ae672e872 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:51.719823 (+170821us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2178386816}
I20260810 18:51:51.747517 16695 leader_election.cc:302] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:51.577230 18859 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:51.668772 16496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1252) took 2104 ms. Trace:
I20260810 18:51:51.549136 18922 leader_election.cc:288] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:51.732163 16508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1272) took 1229 ms. Trace:
W20260810 18:51:51.782125 16573 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.012s sys 0.012s
I20260810 18:51:51.654625 19089 raft_consensus.cc:2726] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 18:51:51.787628 16573 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.565s user 0.012s sys 0.012s
I20260810 18:51:51.579363 18870 leader_election.cc:288] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:51.797217 16448 leader_election.cc:302] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:51.674434 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:51.747128 16505 raft_consensus.cc:1226] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3. Ops: []
W20260810 18:51:51.814455 16506 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52338 (request call id 1219) took 1417 ms (client timeout 1742 ms). Trace:
I20260810 18:51:51.815112 16505 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1271) took 1312 ms. Trace:
W20260810 18:51:51.814905 16506 rpcz_store.cc:269] 0810 18:51:50.397120 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:50.397272 (+ 152us) service_pool.cc:227] Handling call
0810 18:51:51.814270 (+1416998us) spinlock_profiling.cc:243] Waited 186 ms on lock 0x7b54001e14d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:51.814378 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":426877696}
I20260810 18:51:51.815622 16505 rpcz_store.cc:276] 0810 18:51:50.502800 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:51.732281 (+1229481us) service_pool.cc:227] Handling call
0810 18:51:51.732588 (+ 307us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:51.814840 (+ 82252us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:51.815041 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:51:51.816183 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:51.638024 18989 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:51.609193 17770 raft_consensus.cc:3037] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:51.558317 18809 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:51.813784 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:51.826740 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:51.830775 16506 raft_consensus.cc:2454] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:51:51.786374 18948 raft_consensus.cc:482] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
W20260810 18:51:51.844908 18942 consensus_meta.cc:287] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.576s user 0.012s sys 0.004s
W20260810 18:51:51.654191 18883 consensus_meta.cc:287] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.783s user 0.012s sys 0.004s
W20260810 18:51:51.736708 17726 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.000s sys 0.016s
W20260810 18:51:51.846979 17726 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.632s user 0.000s sys 0.024s
I20260810 18:51:51.796008 16750 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6fa467f263e640b095b31b6d998258db"
dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
mode: GRACEFUL
new_leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
from {username='slave'} at 127.0.0.1:45870
I20260810 18:51:51.850744 16750 raft_consensus.cc:595] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Received request to transfer leadership to TS 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:51.633916 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:51.854192 16500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1276) took 1318 ms. Trace:
I20260810 18:51:51.782650 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:51.645635 18970 leader_election.cc:288] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:51.737195 18906 raft_consensus.cc:2726] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:51.793891 19099 consensus_queue.cc:1046] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20260810 18:51:51.582010 18827 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:51.557983 18567 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:51.854763 16500 rpcz_store.cc:276] 0810 18:51:50.536082 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:51.852984 (+1316902us) service_pool.cc:227] Handling call
0810 18:51:51.853465 (+ 481us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:51.853715 (+ 250us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:51.853760 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:51.853832 (+ 72us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:51:51.853882 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:51.853962 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:51.854113 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:51:51.857131 18687 raft_consensus.cc:3032] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:51.802278 18502 raft_consensus.cc:2781] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:51.857788 18687 raft_consensus.cc:729] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:51.858284 18567 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:51.858502 18656 raft_consensus.cc:482] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:51.856210 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:51.859115 18656 raft_consensus.cc:504] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:51.860121 17787 raft_consensus.cc:2454] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:51:51.861971 17726 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:51:51.806034 19047 consensus_meta.cc:287] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.891s user 0.011s sys 0.004s
I20260810 18:51:51.864553 18859 raft_consensus.cc:2726] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:51.865016 19047 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:51.576568 19066 leader_election.cc:288] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:51.666184 19017 leader_election.cc:288] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:51.817397 16447 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:51.868501 18730 raft_consensus.cc:2726] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:51.866875 17726 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:51.787086 18516 raft_consensus.cc:2781] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:51.809789 19092 leader_election.cc:288] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:51.726250 18937 leader_election.cc:288] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:51.869757 18516 raft_consensus.cc:482] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:51.825739 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:51.870498 18516 raft_consensus.cc:3037] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
W20260810 18:51:51.788460 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 565796us
I20260810 18:51:51.824584 16575 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:51.832058 16695 leader_election.cc:302] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:51:51.875594 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 43632.6us
I20260810 18:51:51.876049 19066 raft_consensus.cc:2781] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:51:51.876885 19066 raft_consensus.cc:482] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:51.877499 19066 raft_consensus.cc:3037] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:51.877636 16695 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:51.878856 18827 raft_consensus.cc:2726] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260810 18:51:51.873652 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 49177.9us
I20260810 18:51:51.798619 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:51.882066 17726 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:51.857398 16752 raft_consensus.cc:1226] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:51.857928 18502 raft_consensus.cc:482] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:51.729434 18858 consensus_queue.cc:1046] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.078s
I20260810 18:51:51.614432 16748 raft_consensus.cc:2454] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:51.859395 18687 consensus_queue.cc:260] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:51.784766 16496 rpcz_store.cc:276] 0810 18:51:49.564706 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:50.019887 (+455181us) service_pool.cc:227] Handling call
0810 18:51:51.667853 (+1647966us) spinlock_profiling.cc:243] Waited 3.35 ms on lock 0x7b5800001898. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed0502b0 00007f1aed03b7f9 00007f1aed03b2a2 00007f1ae6721684
0810 18:51:51.668083 (+ 230us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:51.668275 (+ 192us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:51.668323 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:51.668372 (+ 49us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:51:51.668444 (+ 72us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:51.668515 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:51.668701 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7702912}
I20260810 18:51:51.867235 16624 raft_consensus.cc:1226] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 1. Current term is 2. Ops: []
W20260810 18:51:51.868673 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 51349.8us
I20260810 18:51:51.917397 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:51.816690 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:51.609845 18921 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:51.870687 18859 raft_consensus.cc:2726] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:51.788946 16508 rpcz_store.cc:276] 0810 18:51:50.503076 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:51.731373 (+1228297us) service_pool.cc:227] Handling call
0810 18:51:51.731676 (+ 303us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:51.731844 (+ 168us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:51.731866 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:51.731909 (+ 43us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:51:51.731938 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:51.732001 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:51.732115 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:51:51.874058 18937 raft_consensus.cc:2726] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:51:51.903486 16695 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:51.883925 16506 raft_consensus.cc:1226] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:51.819568 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:51:51.668072 19039 consensus_meta.cc:287] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.741s user 0.007s sys 0.009s
I20260810 18:51:51.909176 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:51.870163 18942 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:51.909868 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:51.910187 17726 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:51.823973 17778 raft_consensus.cc:2454] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:51.917976 16447 leader_election.cc:302] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:51.935803 16622 raft_consensus.cc:3037] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:51.936563 16447 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:51.911567 17771 raft_consensus.cc:2454] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:51.862846 18656 leader_election.cc:288] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:51.904538 17780 raft_consensus.cc:1226] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:51.920979 16573 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:51.923132 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:51.904934 18871 leader_election.cc:288] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:51.929198 16506 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1275) took 1415 ms. Trace:
I20260810 18:51:51.953155 16506 rpcz_store.cc:276] 0810 18:51:50.513321 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:51.880173 (+1366852us) service_pool.cc:227] Handling call
0810 18:51:51.880499 (+ 326us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:51.928953 (+ 48454us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:51.929138 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:51:51.911427 18502 raft_consensus.cc:3037] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:51.911449 19029 raft_consensus.cc:2726] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260810 18:51:51.931407 18808 consensus_meta.cc:287] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.526s user 0.008s sys 0.004s
I20260810 18:51:51.961930 18808 raft_consensus.cc:504] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:51.962283 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:51.964116 18808 leader_election.cc:288] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:51.825124 18989 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:51:51.933254 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 48000.7us
I20260810 18:51:51.932986 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:51.873091 18883 raft_consensus.cc:504] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:51.990957 18883 leader_election.cc:288] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:51.991523 18883 raft_consensus.cc:482] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:51.992122 18883 raft_consensus.cc:504] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:51.993366 18957 consensus_meta.cc:287] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.567s user 0.006s sys 0.003s
I20260810 18:51:51.993778 18957 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:51.994817 16631 raft_consensus.cc:1226] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 2. Current term is 3. Ops: []
I20260810 18:51:51.953840 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:51.940225 18859 raft_consensus.cc:2726] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:51.996073 16506 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1230) took 1056 ms. Trace:
I20260810 18:51:51.996448 16506 rpcz_store.cc:276] 0810 18:51:50.939743 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:51.965912 (+1026169us) service_pool.cc:227] Handling call
0810 18:51:51.995651 (+ 29739us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:51.995800 (+ 149us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:51.995824 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:51.995859 (+ 35us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:51:51.995883 (+ 24us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:51.995930 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:51.996024 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 18:51:51.997642 18938 consensus_meta.cc:287] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.920s user 0.005s sys 0.009s
I20260810 18:51:51.998076 16506 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1231) took 1058 ms. Trace:
I20260810 18:51:51.965878 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:51.883576 18948 raft_consensus.cc:504] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:51.967530 16573 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:51.832432 19089 consensus_queue.cc:1046] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.116s
I20260810 18:51:51.937108 18567 raft_consensus.cc:2781] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:51:51.933131 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:51.998023 18938 raft_consensus.cc:504] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:51.999922 16506 rpcz_store.cc:276] 0810 18:51:50.940015 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:51.997274 (+1057259us) service_pool.cc:227] Handling call
0810 18:51:51.997640 (+ 366us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:51.997795 (+ 155us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:51.997819 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:51.997854 (+ 35us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:51:51.997883 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:51.997932 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:51.998027 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:51:52.018123 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:51.966184 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:52.019194 16747 raft_consensus.cc:2454] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 1.
I20260810 18:51:51.939664 18730 raft_consensus.cc:2781] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:51.943499 18982 raft_consensus.cc:2726] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:51:51.943177 19039 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:52.008807 17785 raft_consensus.cc:2354] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 2. Current term is 3.
I20260810 18:51:51.944013 18687 raft_consensus.cc:3037] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
W20260810 18:51:52.006860 19107 log.cc:924] Time spent T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.074s user 0.001s sys 0.000s
I20260810 18:51:52.022452 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:52.006791 18975 raft_consensus.cc:2726] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:52.025182 16622 raft_consensus.cc:2454] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 2.
I20260810 18:51:52.026758 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:52.026985 16745 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3"
dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
mode: GRACEFUL
new_leader_uuid: "831925dd8b644ebab7eb0d672a94eddd"
from {username='slave'} at 127.0.0.1:45870
I20260810 18:51:52.031558 16745 raft_consensus.cc:595] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Received request to transfer leadership to TS 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:52.031628 16636 raft_consensus.cc:1226] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:52.028107 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:51:52.021121 16701 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.038s sys 0.046s
W20260810 18:51:52.032786 16701 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.569s user 0.038s sys 0.046s
I20260810 18:51:51.999500 18970 consensus_queue.cc:1046] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260810 18:51:52.045655 16506 raft_consensus.cc:1226] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:52.035920 16573 leader_election.cc:398] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Vote denied by peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387) with higher term. Message: Invalid argument: T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 2. Current term is 3.
I20260810 18:51:52.046321 16573 leader_election.cc:401] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260810 18:51:52.014789 19056 raft_consensus.cc:2781] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260810 18:51:52.033375 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 570460us
I20260810 18:51:52.013988 18567 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:52.029346 18730 raft_consensus.cc:686] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 2 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:52.035872 16496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1278) took 1274 ms. Trace:
I20260810 18:51:52.036150 16740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1295) took 1609 ms. Trace:
I20260810 18:51:52.037846 18957 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:52.058260 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260810 18:51:52.032804 16500 raft_consensus.cc:2379] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 2: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:52.041320 19039 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:52.042804 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:51:52.044109 18874 consensus_meta.cc:287] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.731s user 0.012s sys 0.001s
I20260810 18:51:52.052349 16701 leader_election.cc:302] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:52.021397 16632 raft_consensus.cc:1226] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:52.034224 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:52.069424 16747 raft_consensus.cc:3037] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
W20260810 18:51:52.069624 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 131977us
I20260810 18:51:52.066308 16628 raft_consensus.cc:2379] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 2: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:52.026679 19018 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:52.069875 16701 leader_election.cc:302] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:52.056634 16506 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1232) took 1116 ms. Trace:
I20260810 18:51:52.076231 16506 rpcz_store.cc:276] 0810 18:51:50.940268 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:52.045145 (+1104877us) service_pool.cc:227] Handling call
0810 18:51:52.045455 (+ 310us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:52.056302 (+ 10847us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:52.056563 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:51:52.020967 19017 consensus_queue.cc:1046] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20260810 18:51:52.084805 16626 raft_consensus.cc:1226] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:52.083135 16447 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:52.086278 16447 leader_election.cc:302] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:51:52.087006 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 53391.4us
I20260810 18:51:52.059087 18567 raft_consensus.cc:3037] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:52.054618 19056 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:51:52.089164 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:52.091387 16626 raft_consensus.cc:1226] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:52.090848 16701 leader_election.cc:302] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:52.089632 16447 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:52.068336 17781 raft_consensus.cc:1226] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:52.071163 18809 raft_consensus.cc:2726] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:51:52.101975 16701 leader_election.cc:302] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:52.072386 19047 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:52.100093 16747 raft_consensus.cc:2454] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:52.057858 16740 rpcz_store.cc:276] 0810 18:51:50.426819 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:50.638043 (+211224us) service_pool.cc:227] Handling call
0810 18:51:50.638354 (+ 311us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:51.318697 (+680343us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:52.032738 (+714041us) spinlock_profiling.cc:243] Waited 627 ms on lock 0x7b54002c2ed4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:51:52.036089 (+ 3351us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1441465728}
I20260810 18:51:52.098202 18502 raft_consensus.cc:504] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:52.103930 19111 raft_consensus.cc:2726] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:52.109752 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:51:52.114668 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 175263us
I20260810 18:51:52.115361 17726 leader_election.cc:302] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:52.097504 19077 raft_consensus.cc:2781] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:52.110318 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:52.088840 18872 raft_consensus.cc:2726] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260810 18:51:52.093146 18862 consensus_meta.cc:287] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.764s user 0.007s sys 0.007s
I20260810 18:51:52.136119 18862 raft_consensus.cc:504] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:52.136788 17770 raft_consensus.cc:2454] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
I20260810 18:51:52.137850 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:52.054481 17727 leader_election.cc:302] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:52.104113 18858 raft_consensus.cc:2781] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:51:52.139387 18858 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:51:52.057835 16496 rpcz_store.cc:276] 0810 18:51:50.761789 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:51.925181 (+1163392us) service_pool.cc:227] Handling call
0810 18:51:51.955953 (+ 30772us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:52.011105 (+ 55152us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:52.013017 (+ 1912us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:52.019175 (+ 6158us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:51:52.023827 (+ 4652us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:52.028927 (+ 5100us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:52.035771 (+ 6844us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:51:52.139954 18858 raft_consensus.cc:3037] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
W20260810 18:51:52.141196 18905 consensus_meta.cc:287] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.923s user 0.010s sys 0.001s
I20260810 18:51:52.141899 18905 raft_consensus.cc:2726] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:51:52.099243 16626 raft_consensus.cc:1226] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3. Ops: []
I20260810 18:51:52.142949 16622 raft_consensus.cc:1226] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3. Ops: []
I20260810 18:51:52.143527 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:52.143814 16502 raft_consensus.cc:2379] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 2: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:52.144363 16626 raft_consensus.cc:2379] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 2: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:52.144923 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:52.145620 16626 raft_consensus.cc:1226] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260810 18:51:52.146006 16501 raft_consensus.cc:1226] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 2. Current term is 3. Ops: []
I20260810 18:51:52.119614 18502 leader_election.cc:288] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:52.145323 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:52.092415 18970 raft_consensus.cc:3037] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:52.104892 16701 leader_election.cc:302] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:52.110339 18871 raft_consensus.cc:2726] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:52.119426 16505 raft_consensus.cc:1226] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:52.122943 16450 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:52.092378 19056 raft_consensus.cc:3037] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:52.124895 17778 raft_consensus.cc:1226] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 3. Ops: []
I20260810 18:51:52.125805 18948 leader_election.cc:288] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:52.166632 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:52.168316 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:51:52.126071 19078 log.cc:924] Time spent T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 1.101s user 0.003s sys 0.010s
W20260810 18:51:52.126673 19110 log.cc:924] Time spent T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260810 18:51:52.070953 18874 raft_consensus.cc:504] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:52.171514 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 66739.2us
I20260810 18:51:52.171842 18874 leader_election.cc:288] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:52.127314 16621 raft_consensus.cc:1226] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 3. Current term is 4. Ops: []
I20260810 18:51:52.172276 16701 leader_election.cc:302] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:52.172744 16508 raft_consensus.cc:1226] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 2. Current term is 3. Ops: []
I20260810 18:51:52.173305 16508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1229) took 1233 ms. Trace:
I20260810 18:51:52.173672 16701 leader_election.cc:302] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:52.173599 16508 rpcz_store.cc:276] 0810 18:51:50.939446 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:52.172297 (+1232851us) service_pool.cc:227] Handling call
0810 18:51:52.172589 (+ 292us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:52.173102 (+ 513us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:52.173258 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:51:52.174678 19089 raft_consensus.cc:2726] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:52.127935 18942 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:52.128556 19018 raft_consensus.cc:2781] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260810 18:51:52.176317 19050 consensus_meta.cc:287] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 1.023s user 0.013s sys 0.000s
I20260810 18:51:52.176519 19018 raft_consensus.cc:482] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:52.176976 19018 raft_consensus.cc:3037] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:52.124243 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:52.177289 16701 leader_election.cc:302] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:52.106143 18989 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:52.138916 16632 raft_consensus.cc:2379] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 2: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:52.184082 16623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1269) took 1871 ms. Trace:
I20260810 18:51:52.150024 17726 leader_election.cc:302] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:51:52.156701 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 127803us
W20260810 18:51:52.157428 18728 consensus_meta.cc:287] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.877s user 0.005s sys 0.008s
I20260810 18:51:52.158419 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:52.145669 17785 raft_consensus.cc:2454] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:52.184507 16623 rpcz_store.cc:276] 0810 18:51:50.312545 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:50.312805 (+ 260us) service_pool.cc:227] Handling call
0810 18:51:50.313200 (+ 395us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:51.080115 (+766915us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:52.183838 (+1103723us) spinlock_profiling.cc:243] Waited 766 ms on lock 0x7b5400367ed4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:51:52.184021 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1761850752}
I20260810 18:51:52.161588 19114 raft_consensus.cc:2781] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:52.162747 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:52.166185 19099 raft_consensus.cc:2726] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:51:52.176169 19089 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:52.112172 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
W20260810 18:51:52.133996 16495 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52352 (request call id 1267) took 1646 ms (client timeout 1613 ms). Trace:
I20260810 18:51:52.108646 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
W20260810 18:51:52.181910 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 92362.2us
W20260810 18:51:52.186336 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 63434.4us
I20260810 18:51:52.187093 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:52.142759 18516 raft_consensus.cc:504] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:52.215718 16450 leader_election.cc:302] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:52.217340 16450 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:52.182832 16695 leader_election.cc:302] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:52.190452 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:52.189113 17783 raft_consensus.cc:1226] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:52.130563 19077 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:52.218389 16626 raft_consensus.cc:3037] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:52.176748 19050 raft_consensus.cc:504] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:52.189455 18538 raft_consensus.cc:2781] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:52.239874 18538 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:52.240455 18538 raft_consensus.cc:3037] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:52.242895 18809 raft_consensus.cc:2781] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:52.243732 18809 raft_consensus.cc:686] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:52.244212 19029 consensus_queue.cc:1046] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260810 18:51:52.245591 19029 raft_consensus.cc:3032] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:51:52.245378 18809 consensus_queue.cc:237] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:52.246179 19029 raft_consensus.cc:729] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:51:52.204715 17781 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55570 (request call id 320) took 1293 ms. Trace:
I20260810 18:51:52.188933 18948 raft_consensus.cc:2726] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:52.236270 19077 raft_consensus.cc:3037] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:52.226980 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:52.264819 18937 raft_consensus.cc:979] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb: : Instructing follower d304ac47fdf846de81337057a4ef7bea to start an election
I20260810 18:51:52.192411 18906 raft_consensus.cc:2726] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 18:51:52.230648 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 47992.5us
I20260810 18:51:52.265349 18937 raft_consensus.cc:1067] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Signalling peer d304ac47fdf846de81337057a4ef7bea to start an election
I20260810 18:51:52.194516 18862 leader_election.cc:288] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:52.265092 19050 leader_election.cc:288] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:52.123250 18730 consensus_queue.cc:237] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:52.258514 17781 rpcz_store.cc:276] 0810 18:51:50.911668 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:50.915181 (+ 3513us) service_pool.cc:227] Handling call
0810 18:51:50.939624 (+ 24443us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:52.100679 (+1161055us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:52.178619 (+ 77940us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54006a27d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:51:52.196232 (+ 17613us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2546575616}
I20260810 18:51:52.205598 16740 raft_consensus.cc:3037] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:52.271651 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:51:52.272539 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:51:52.196308 16500 raft_consensus.cc:1226] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:52.280064 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:52.280488 16626 raft_consensus.cc:2454] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:52.228480 18981 raft_consensus.cc:2781] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:52.219120 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:52.184598 18883 leader_election.cc:288] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:52.195564 19114 raft_consensus.cc:482] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:52.230010 18938 leader_election.cc:288] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:52.247546 19105 raft_consensus.cc:2741] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
W20260810 18:51:52.209241 16495 rpcz_store.cc:269] 0810 18:51:50.487638 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:50.580074 (+ 92436us) service_pool.cc:227] Handling call
0810 18:51:52.133666 (+1553592us) spinlock_profiling.cc:243] Waited 832 ms on lock 0x7b54000f6754. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:52.133898 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1913690496}
I20260810 18:51:52.275779 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:52.277127 16573 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:51:52.283011 16448 leader_election.cc:302] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:52.267055 18957 raft_consensus.cc:3032] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:52.290489 16342 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55260 (request call id 121) took 5379 ms. Trace:
I20260810 18:51:52.228829 19065 raft_consensus.cc:2726] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260810 18:51:52.195048 18728 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:52.312299 17786 raft_consensus.cc:1226] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 3. Current term is 4. Ops: []
W20260810 18:51:52.312892 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 105141us
I20260810 18:51:52.308873 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:52.310626 16342 rpcz_store.cc:276] 0810 18:51:46.896857 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:46.897026 (+ 169us) service_pool.cc:227] Handling call
0810 18:51:52.275968 (+5378942us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 18:51:49.272053 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 18:51:49.419264 (+147211us) write_op.cc:432] Acquiring schema lock in shared mode
0810 18:51:49.420496 (+ 1232us) write_op.cc:435] Acquired schema lock
0810 18:51:49.420526 (+ 30us) tablet.cc:660] Decoding operations
0810 18:51:49.454303 (+ 33777us) write_op.cc:620] Acquiring the partition lock for write op
0810 18:51:49.464608 (+ 10305us) write_op.cc:641] Partition lock acquired for write op
0810 18:51:49.477919 (+ 13311us) tablet.cc:683] Acquiring locks for 1 operations
0810 18:51:49.539729 (+ 61810us) tablet.cc:699] Row locks acquired
0810 18:51:49.539758 (+ 29us) write_op.cc:260] PREPARE: finished
0810 18:51:49.711043 (+171285us) write_op.cc:270] Start()
0810 18:51:49.728373 (+ 17330us) write_op.cc:276] Timestamp: P: 1786387909703802 usec, L: 0
0810 18:51:49.728397 (+ 24us) op_driver.cc:331] REPLICATION: starting
0810 18:51:49.781034 (+ 52637us) log.cc:842] Serialized 536 byte log entry
0810 18:51:50.895389 (+1114355us) op_driver.cc:444] REPLICATION: finished
0810 18:51:51.186353 (+290964us) write_op.cc:301] APPLY: starting
0810 18:51:51.197671 (+ 11318us) tablet.cc:1365] starting BulkCheckPresence
0810 18:51:51.213098 (+ 15427us) tablet.cc:1368] finished BulkCheckPresence
0810 18:51:51.250458 (+ 37360us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 18:51:51.272181 (+ 21723us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 18:51:51.285947 (+ 13766us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 18:51:51.299160 (+ 13213us) write_op.cc:312] APPLY: finished
0810 18:51:51.335780 (+ 36620us) log.cc:842] Serialized 23 byte log entry
0810 18:51:51.366754 (+ 30974us) write_op.cc:489] Releasing partition, row and schema locks
0810 18:51:51.387289 (+ 20535us) write_op.cc:454] Released schema lock
0810 18:51:51.399185 (+ 11896us) write_op.cc:341] FINISH: Updating metrics
0810 18:51:51.420740 (+ 21555us) write_op.cc:489] Releasing partition, row and schema locks
0810 18:51:51.422924 (+ 2184us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":976054,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":235618,"apply.run_cpu_time_us":4152,"apply.run_wall_time_us":260013,"num_ops":1,"prepare.queue_time_us":1240940,"prepare.run_cpu_time_us":92229,"prepare.run_wall_time_us":695257,"raft.queue_time_us":314997,"raft.run_cpu_time_us":38082,"raft.run_wall_time_us":207721,"replication_time_us":1158836,"thread_start_us":1450871,"threads_started":4,"wal-append.queue_time_us":172185}]]}
I20260810 18:51:52.339226 17770 raft_consensus.cc:1226] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:52.343127 16701 leader_election.cc:302] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:52.306548 18981 raft_consensus.cc:482] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:52.308411 19114 raft_consensus.cc:3037] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:52.354586 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:52.311633 16621 raft_consensus.cc:3037] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
W20260810 18:51:52.354316 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 71448.4us
I20260810 18:51:52.344627 18946 raft_consensus.cc:2781] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:52.367148 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:52.327553 19029 consensus_queue.cc:260] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:52.310504 18957 raft_consensus.cc:729] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
W20260810 18:51:52.354934 18941 consensus_meta.cc:287] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.951s user 0.009s sys 0.006s
I20260810 18:51:52.285112 18580 consensus_queue.cc:1046] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.076s
I20260810 18:51:52.307807 18687 consensus_queue.cc:1046] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
W20260810 18:51:52.377146 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 84540.5us
W20260810 18:51:52.378396 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 151603us
I20260810 18:51:52.367225 18981 raft_consensus.cc:3037] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:52.391593 16450 leader_election.cc:302] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:51:52.359741 17775 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55570 (request call id 313) took 1647 ms (client timeout 2173 ms). Trace:
W20260810 18:51:52.376408 19073 consensus_meta.cc:287] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.881s user 0.009s sys 0.007s
I20260810 18:51:52.367547 19125 raft_consensus.cc:2781] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:52.371271 18946 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:52.359728 19099 raft_consensus.cc:2726] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:52.400449 18946 raft_consensus.cc:3037] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:52.400274 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:52.402936 18728 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:52.406981 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:52.378580 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:52.428509 17724 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:52.378024 18870 consensus_queue.cc:1046] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260810 18:51:52.430923 17724 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:52.389845 18941 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:52.444006 16450 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:52.422338 18948 raft_consensus.cc:2726] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 18:51:52.436149 19066 consensus_meta.cc:287] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.558s user 0.011s sys 0.004s
W20260810 18:51:52.448423 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 81616.1us
I20260810 18:51:52.395340 19065 consensus_queue.cc:1035] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.040s
I20260810 18:51:52.405900 19073 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:52.434325 19125 raft_consensus.cc:686] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 2 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
W20260810 18:51:52.392952 17775 rpcz_store.cc:269] 0810 18:51:50.695977 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:50.703356 (+ 7379us) service_pool.cc:227] Handling call
0810 18:51:52.310694 (+1607338us) spinlock_profiling.cc:243] Waited 668 ms on lock 0x7b54004f06d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:52.343088 (+ 32394us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1535513472}
I20260810 18:51:52.416671 19111 consensus_queue.cc:1046] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260810 18:51:52.471387 16507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1223) took 2073 ms. Trace:
I20260810 18:51:52.471876 16507 rpcz_store.cc:276] 0810 18:51:50.398465 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:50.743186 (+344721us) service_pool.cc:227] Handling call
0810 18:51:51.550759 (+807573us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:51.550977 (+ 218us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:51.551011 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:51.551079 (+ 68us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:51:51.551114 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:51.551165 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:52.471176 (+920011us) spinlock_profiling.cc:243] Waited 807 ms on lock 0x7b54000f6250. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:52.471318 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1856295808}
I20260810 18:51:52.456728 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:52.436553 19056 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:52.489392 16450 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:52.439160 19068 consensus_queue.cc:1046] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20260810 18:51:52.467006 18982 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:52.453840 19066 raft_consensus.cc:504] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:52.481722 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:52.445804 18957 consensus_queue.cc:260] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:52.504408 18982 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:52.516875 18567 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:52.479707 17780 raft_consensus.cc:2932] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:52.527050 16749 raft_consensus.cc:2379] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 4: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:51:52.514611 19151 raft_consensus.cc:2726] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:51:52.466555 18808 consensus_queue.cc:1046] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20260810 18:51:52.466393 19149 consensus_queue.cc:1046] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
I20260810 18:51:52.466537 18687 consensus_queue.cc:1046] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260810 18:51:52.499097 19141 raft_consensus.cc:2726] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:51:52.504370 19092 raft_consensus.cc:3037] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
W20260810 18:51:52.519354 18949 consensus_meta.cc:287] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 1.089s user 0.005s sys 0.011s
I20260810 18:51:52.536996 19147 consensus_queue.cc:1046] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20260810 18:51:52.523294 16496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1285) took 1092 ms. Trace:
I20260810 18:51:52.550014 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
W20260810 18:51:52.553287 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 64025.7us
I20260810 18:51:52.554858 16450 leader_election.cc:302] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:52.556353 16450 leader_election.cc:302] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:52.558051 16450 leader_election.cc:302] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:52.559315 16450 leader_election.cc:302] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:51:52.572738 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 141947us
I20260810 18:51:52.546250 18538 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:52.576678 17724 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:52.536132 18858 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:52.591081 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:52.592589 16450 leader_election.cc:302] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:52.564246 18728 raft_consensus.cc:2781] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:52.595929 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:52.601044 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:52.602444 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:52.582772 18948 raft_consensus.cc:2726] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:52.606367 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:52.547947 18949 raft_consensus.cc:504] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:52.610741 19131 log.cc:924] Time spent T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.095s user 0.001s sys 0.000s
I20260810 18:51:52.614151 16742 raft_consensus.cc:2454] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:52.616225 18982 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:52.546880 16496 rpcz_store.cc:276] 0810 18:51:51.427777 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:52.140535 (+712758us) service_pool.cc:227] Handling call
0810 18:51:52.141123 (+ 588us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:52.141470 (+ 347us) raft_consensus.cc:1520] Early marking committed up to index 2
0810 18:51:52.304352 (+162882us) log.cc:842] Serialized 14 byte log entry
0810 18:51:52.484813 (+180461us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:52.488413 (+ 3600us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:51:52.493429 (+ 5016us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:52.503418 (+ 9989us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:52.519714 (+ 16296us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41196,"prepare.queue_time_us":49809,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":27,"raft.queue_time_us":49959,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":7170,"spinlock_wait_cycles":25929344,"thread_start_us":224763,"threads_started":2,"wal-append.queue_time_us":176151}
I20260810 18:51:52.594252 18970 raft_consensus.cc:2726] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387) with higher term. Message: Invalid argument: T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 2. Current term is 3.
I20260810 18:51:52.593122 18941 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:52.550943 19015 consensus_queue.cc:1046] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.175s
W20260810 18:51:52.629933 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 162581us
I20260810 18:51:52.618769 18948 raft_consensus.cc:2726] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:51:52.597487 18870 raft_consensus.cc:2726] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:51:52.534195 19125 consensus_queue.cc:237] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:52.644477 16628 raft_consensus.cc:1226] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 3. Ops: []
I20260810 18:51:52.609176 18728 raft_consensus.cc:686] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 2 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:52.607630 19141 raft_consensus.cc:2726] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:51:52.641371 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:52.623303 19073 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:52.650115 16633 raft_consensus.cc:2379] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 3: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:52.652303 16747 raft_consensus.cc:1226] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: []
I20260810 18:51:52.656409 18921 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:52.658704 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:52.660543 16505 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1277) took 2120 ms. Trace:
I20260810 18:51:52.660893 16505 rpcz_store.cc:276] 0810 18:51:50.540382 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:51.919356 (+1378974us) service_pool.cc:227] Handling call
0810 18:51:51.919699 (+ 343us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:52.164366 (+244667us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:52.660333 (+495967us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x7b54001cb1d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:51:52.660491 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":458946944}
I20260810 18:51:52.665489 16743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43726 (request call id 1263) took 1222 ms. Trace:
I20260810 18:51:52.665854 16743 rpcz_store.cc:276] 0810 18:51:51.443538 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:51.443748 (+ 210us) service_pool.cc:227] Handling call
0810 18:51:52.665263 (+1221515us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b540020b054. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:52.665412 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":233751296}
I20260810 18:51:52.667164 16750 raft_consensus.cc:1226] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:52.667686 18567 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:52.668740 16743 raft_consensus.cc:1226] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 3. Current term is 4. Ops: []
I20260810 18:51:52.647291 17770 raft_consensus.cc:1226] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 3. Ops: []
I20260810 18:51:52.645044 19056 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:51:52.681713 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 44799.8us
I20260810 18:51:52.676115 18858 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:52.675779 18728 consensus_queue.cc:237] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:52.677737 16636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1272) took 1362 ms. Trace:
I20260810 18:51:52.684031 16636 rpcz_store.cc:276] 0810 18:51:51.315293 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:51.421298 (+106005us) service_pool.cc:227] Handling call
0810 18:51:51.433011 (+ 11713us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:52.032184 (+599173us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:52.677541 (+645357us) spinlock_profiling.cc:243] Waited 598 ms on lock 0x7b54002e1654. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:51:52.677683 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1376111488}
I20260810 18:51:52.669829 18870 raft_consensus.cc:482] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:52.679764 16500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1283) took 1366 ms. Trace:
I20260810 18:51:52.616273 19152 raft_consensus.cc:2726] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:51:52.661824 19066 leader_election.cc:288] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:52.667732 19160 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:52.692843 16633 raft_consensus.cc:1226] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 2. Current term is 3. Ops: []
I20260810 18:51:52.680042 19155 consensus_queue.cc:1046] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260810 18:51:52.677651 18538 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:52.700023 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:52.702548 16450 leader_election.cc:302] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:52.706028 19056 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:52.683631 16740 raft_consensus.cc:2454] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
W20260810 18:51:52.701566 16748 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43726 (request call id 1266) took 1256 ms (client timeout 1573 ms). Trace:
W20260810 18:51:52.712455 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 86918us
I20260810 18:51:52.713943 16450 leader_election.cc:302] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:52.715940 16575 leader_election.cc:302] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:52.717525 16575 leader_election.cc:302] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:52.718480 16575 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:52.719082 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:52.691588 18870 raft_consensus.cc:504] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:52.721271 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:52.723224 19047 raft_consensus.cc:2726] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260810 18:51:52.727586 19153 log.cc:924] Time spent T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.116s user 0.001s sys 0.000s
I20260810 18:51:52.700862 19160 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:52.723663 18538 raft_consensus.cc:3037] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:52.686494 19105 consensus_queue.cc:1046] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20260810 18:51:52.700892 19066 consensus_queue.cc:1046] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260810 18:51:52.771363 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:52.723992 18921 raft_consensus.cc:2726] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:52.771426 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:52.778115 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 59684.2us
I20260810 18:51:52.724563 19152 raft_consensus.cc:3037] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
W20260810 18:51:52.783504 19157 log.cc:924] Time spent T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.068s user 0.000s sys 0.001s
I20260810 18:51:52.730816 18970 raft_consensus.cc:2781] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:51:52.687208 16500 rpcz_store.cc:276] 0810 18:51:51.313114 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:52.136862 (+823748us) service_pool.cc:227] Handling call
0810 18:51:52.137328 (+ 466us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:52.286393 (+149065us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:52.679382 (+392989us) spinlock_profiling.cc:243] Waited 41.3 ms on lock 0x7b54000e10d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:51:52.679661 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":94999296}
I20260810 18:51:52.723989 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:52.710590 18858 consensus_queue.cc:1046] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.054s
I20260810 18:51:52.734241 18870 leader_election.cc:288] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:52.704821 19141 consensus_queue.cc:1046] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.088s
I20260810 18:51:52.733871 19111 consensus_queue.cc:1046] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.054s
I20260810 18:51:52.740217 19154 raft_consensus.cc:2726] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:51:52.792666 18970 raft_consensus.cc:482] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:52.777213 19160 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:52.787890 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:52.796900 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:52.714617 19056 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:52.784017 19156 consensus_queue.cc:1046] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.068s
I20260810 18:51:52.723942 19129 consensus_queue.cc:1046] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.207s
W20260810 18:51:52.714552 16748 rpcz_store.cc:269] 0810 18:51:51.444688 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:51.446484 (+ 1796us) service_pool.cc:227] Handling call
0810 18:51:52.701371 (+1254887us) spinlock_profiling.cc:243] Waited 86.6 ms on lock 0x7b54002391d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:52.701500 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":199218944}
I20260810 18:51:52.793835 18687 consensus_queue.cc:1046] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20260810 18:51:52.803743 19099 raft_consensus.cc:3032] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:52.795912 18827 raft_consensus.cc:3032] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:51:52.806357 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:52.836707 18827 raft_consensus.cc:729] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:52.807374 16505 raft_consensus.cc:1226] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 2. Current term is 3. Ops: []
I20260810 18:51:52.839577 18827 consensus_queue.cc:260] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:52.793279 18949 leader_election.cc:288] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:52.829932 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:52.841028 18827 raft_consensus.cc:3037] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:52.809244 19162 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:52.836792 19099 raft_consensus.cc:729] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:52.798480 18870 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:52.816151 18970 raft_consensus.cc:3037] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:52.837126 17785 raft_consensus.cc:1226] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:52.832652 19114 raft_consensus.cc:504] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:52.838742 16621 raft_consensus.cc:2454] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:52.840003 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:52.830721 19156 raft_consensus.cc:3032] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:52.833825 19056 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:52.798115 16495 raft_consensus.cc:2454] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:52.855907 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:52.847829 18870 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:52.847926 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:52.858098 16501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1224) took 2459 ms. Trace:
I20260810 18:51:52.859999 16501 rpcz_store.cc:276] 0810 18:51:50.398770 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:50.772479 (+373709us) service_pool.cc:227] Handling call
0810 18:51:52.145827 (+1373348us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:52.159320 (+ 13493us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:52.857857 (+698537us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b54000f6750. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:52.858028 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3157652736}
I20260810 18:51:52.859081 16505 raft_consensus.cc:2379] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 2: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:52.864818 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:52.865298 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:52.866616 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:52.843052 19162 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:52.855260 19156 raft_consensus.cc:729] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
W20260810 18:51:52.914189 19077 consensus_meta.cc:287] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.651s user 0.005s sys 0.010s
I20260810 18:51:52.873835 19164 raft_consensus.cc:3032] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:51:52.868042 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:52.878389 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:52.874750 19056 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:51:52.882846 16743 raft_consensus.cc:1226] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:52.875609 17781 raft_consensus.cc:1226] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:52.852394 18981 raft_consensus.cc:504] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:52.889704 18870 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:52.931161 19158 raft_consensus.cc:3032] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:51:52.931496 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:52.932302 18941 raft_consensus.cc:3032] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:52.932734 18941 raft_consensus.cc:729] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:52.932798 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:52.934069 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:52.933728 18941 consensus_queue.cc:260] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:52.910029 19018 consensus_meta.cc:287] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.732s user 0.013s sys 0.004s
I20260810 18:51:52.935451 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:52.890180 19114 leader_election.cc:288] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:52.935592 19018 raft_consensus.cc:504] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:52.929510 19056 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:52.937181 17787 raft_consensus.cc:1226] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:52.895256 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:52.895975 18975 raft_consensus.cc:3032] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:52.941313 16746 raft_consensus.cc:3032] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:52.879709 16500 raft_consensus.cc:3037] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:52.864146 19099 consensus_queue.cc:260] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:52.903626 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:52.904332 19112 raft_consensus.cc:3032] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:52.906121 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:52.885380 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:52.913712 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:52.918794 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:52.921167 16628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1279) took 1421 ms. Trace:
I20260810 18:51:52.923038 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:52.924396 19135 raft_consensus.cc:3032] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:52.915022 19156 consensus_queue.cc:260] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:52.927754 17783 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55570 (request call id 329) took 1253 ms. Trace:
I20260810 18:51:52.927835 18949 raft_consensus.cc:3032] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:52.931607 19158 raft_consensus.cc:729] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:51:52.868801 16504 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1268) took 2380 ms. Trace:
I20260810 18:51:52.941838 16746 raft_consensus.cc:729] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:53.002825 16508 raft_consensus.cc:1226] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:52.948372 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:53.088248 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:53.088493 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:52.940610 16505 raft_consensus.cc:2379] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 2: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:52.938413 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:52.914603 19077 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:53.094950 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:53.098899 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:53.089279 16504 rpcz_store.cc:276] 0810 18:51:50.487961 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:51.373485 (+885524us) service_pool.cc:227] Handling call
0810 18:51:51.806246 (+432761us) spinlock_profiling.cc:243] Waited 17.3 ms on lock 0x7b5800001898. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519ab7c 00007f1ae519a73b 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed0502b0 00007f1aed03b7f9 00007f1aed03b2a2 00007f1ae6721684
0810 18:51:51.806443 (+ 197us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:51:51.806824 (+ 381us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:51.806855 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:51:52.283450 (+476595us) spinlock_profiling.cc:243] Waited 9.93 ms on lock 0x7b2c000307a8. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae6adde11 00007f1ae6aed442 00007f1ae8135c11 00007f1ae80fa58a 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772
0810 18:51:52.340989 (+ 57539us) log.cc:842] Serialized 26 byte log entry
0810 18:51:52.594207 (+253218us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:51:52.594247 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:52.594310 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:51:52.868557 (+274247us) raft_consensus.cc:1671] finished
0810 18:51:52.868582 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:52.868735 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":115,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":134,"spinlock_wait_cycles":64543616,"thread_start_us":228205,"threads_started":1,"wal-append.queue_time_us":294990}
I20260810 18:51:53.099692 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
W20260810 18:51:53.029695 16502 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52338 (request call id 1213) took 3528 ms (client timeout 1534 ms). Trace:
W20260810 18:51:53.075699 19089 consensus_meta.cc:287] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.864s user 0.008s sys 0.007s
I20260810 18:51:53.059290 19093 raft_consensus.cc:979] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be: : Instructing follower 831925dd8b644ebab7eb0d672a94eddd to start an election
I20260810 18:51:53.084479 16628 rpcz_store.cc:276] 0810 18:51:51.499907 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:52.070494 (+570587us) service_pool.cc:227] Handling call
0810 18:51:52.098599 (+ 28105us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:52.646856 (+548257us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:52.920944 (+274088us) spinlock_profiling.cc:243] Waited 541 ms on lock 0x7b540004d8d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:51:52.921109 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1243620224}
I20260810 18:51:53.108723 16505 raft_consensus.cc:1261] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260810 18:51:52.942564 18494 raft_consensus.cc:3032] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:51:53.073918 18538 raft_consensus.cc:2726] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:53.115486 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:53.022729 18941 raft_consensus.cc:3037] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:53.149850 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:53.096370 16748 raft_consensus.cc:1226] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:52.941638 18975 raft_consensus.cc:729] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:53.120282 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:53.160029 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:53.149415 19156 raft_consensus.cc:3037] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:53.119709 19158 consensus_queue.cc:260] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:53.072968 19094 raft_consensus.cc:482] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:52.939097 19164 raft_consensus.cc:729] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:53.087921 18949 raft_consensus.cc:729] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
W20260810 18:51:53.108096 18946 consensus_meta.cc:287] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.683s user 0.007s sys 0.009s
I20260810 18:51:53.160041 16621 raft_consensus.cc:2379] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 3: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:53.102061 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:53.124818 19089 raft_consensus.cc:2726] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:51:53.044076 18970 raft_consensus.cc:504] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:53.044266 19056 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:51:53.106732 16502 rpcz_store.cc:269] 0810 18:51:49.495031 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:49.498044 (+ 3013us) service_pool.cc:227] Handling call
0810 18:51:51.192279 (+1694235us) spinlock_profiling.cc:243] Waited 20.1 ms on lock 0x7b5800001898. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519abce 000000000034aea9 00007f1aecfc6a30 00007f1aed08c7b4 00007f1aed08c821 00007f1aed03c1b0 00007f1aed03ba94 00007f1ae6723c04 00007f1ae6723b92
0810 18:51:53.001205 (+1808926us) spinlock_profiling.cc:243] Waited 879 ms on lock 0x7b5400284e54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:53.023694 (+ 22489us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2066731136}
I20260810 18:51:53.122309 18862 raft_consensus.cc:482] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:53.152386 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:53.097404 19099 raft_consensus.cc:3037] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:53.104401 19112 raft_consensus.cc:729] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:53.102779 19135 raft_consensus.cc:729] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:53.163676 16448 leader_election.cc:302] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:51:53.224395 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 60798.3us
I20260810 18:51:53.044858 18942 consensus_queue.cc:1046] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.111s
I20260810 18:51:53.029356 19050 consensus_queue.cc:1046] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.126s
I20260810 18:51:53.154578 16495 raft_consensus.cc:2379] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 3: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:53.168905 19152 raft_consensus.cc:2726] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:53.124301 18494 raft_consensus.cc:729] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:53.231936 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:53.232277 18494 consensus_queue.cc:260] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:53.224987 19168 consensus_queue.cc:1046] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20260810 18:51:53.235849 18852 raft_consensus.cc:2781] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:53.236586 19168 raft_consensus.cc:3032] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:53.236848 18852 raft_consensus.cc:686] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 2 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:53.237243 19168 raft_consensus.cc:729] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:53.147408 19017 raft_consensus.cc:482] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:53.171957 16447 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:53.175264 16742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43738 (request call id 1308) took 1093 ms. Trace:
I20260810 18:51:53.162770 18567 raft_consensus.cc:1067] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Signalling peer 831925dd8b644ebab7eb0d672a94eddd to start an election
I20260810 18:51:53.179776 16749 raft_consensus.cc:1226] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:53.240469 18852 consensus_queue.cc:237] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:53.242725 16742 rpcz_store.cc:276] 0810 18:51:52.081552 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:52.084739 (+ 3187us) service_pool.cc:227] Handling call
0810 18:51:53.173572 (+1088833us) spinlock_profiling.cc:243] Waited 75.7 ms on lock 0x7b5400155254. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:53.175185 (+ 1613us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":174216576}
I20260810 18:51:53.152117 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:53.245038 16742 raft_consensus.cc:1226] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 2. Current term is 3. Ops: []
W20260810 18:51:53.245240 19092 consensus_meta.cc:287] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.700s user 0.007s sys 0.008s
I20260810 18:51:53.245887 19092 raft_consensus.cc:2726] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:51:53.162894 19018 leader_election.cc:288] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:53.091162 19097 consensus_queue.cc:1046] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.108s
I20260810 18:51:53.089226 17783 rpcz_store.cc:276] 0810 18:51:51.674716 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:51.737882 (+ 63166us) service_pool.cc:227] Handling call
0810 18:51:52.145718 (+407836us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:52.230229 (+ 84511us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:52.927417 (+697188us) spinlock_profiling.cc:243] Waited 335 ms on lock 0x7b54005e0450. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:52.927684 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":769528704}
W20260810 18:51:53.189453 17780 consensus_meta.cc:287] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.601s user 0.004s sys 0.010s
I20260810 18:51:53.183152 18981 leader_election.cc:288] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:53.043233 19162 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:51:53.250304 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 78388.4us
I20260810 18:51:53.254639 17724 leader_election.cc:302] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:53.233886 16504 raft_consensus.cc:1226] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 2. Current term is 3. Ops: []
I20260810 18:51:53.235924 18494 raft_consensus.cc:3037] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:53.092366 17778 raft_consensus.cc:1799] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Handling vote request from an unknown peer 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:53.178915 18946 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:53.256345 17724 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:53.261893 17724 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:53.267544 16573 leader_election.cc:302] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:53.270423 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:53.271251 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:53.204157 18862 raft_consensus.cc:504] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:53.277258 16573 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:53.270776 19132 raft_consensus.cc:2726] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:51:53.273258 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:53.261806 19039 raft_consensus.cc:2726] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:51:53.283630 16633 raft_consensus.cc:1226] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:53.274151 19114 raft_consensus.cc:2726] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:51:53.262233 19018 raft_consensus.cc:2726] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:53.284307 19068 raft_consensus.cc:2726] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:53.285014 17778 raft_consensus.cc:2379] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 2: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:51:53.283964 16631 raft_consensus.cc:1799] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Handling vote request from an unknown peer d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:53.283102 19047 raft_consensus.cc:2726] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:53.251879 18709 raft_consensus.cc:3032] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:53.287717 16573 leader_election.cc:302] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:53.294204 16631 raft_consensus.cc:2379] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 2: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:53.238960 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:53.273386 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:53.264842 16748 raft_consensus.cc:1226] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 3. Current term is 4. Ops: []
I20260810 18:51:53.304833 16447 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:53.320616 16447 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:53.227043 18975 consensus_queue.cc:260] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:53.240512 19017 raft_consensus.cc:504] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:53.322532 16447 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:53.283999 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:53.296056 16495 raft_consensus.cc:1226] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:53.292276 17775 raft_consensus.cc:1261] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260810 18:51:53.293641 18709 raft_consensus.cc:729] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:53.248615 19164 consensus_queue.cc:260] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:53.178246 19094 raft_consensus.cc:504] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:53.348932 19017 leader_election.cc:288] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:53.328315 19154 raft_consensus.cc:2726] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260810 18:51:53.344733 19152 raft_consensus.cc:2726] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:51:53.322084 16502 raft_consensus.cc:3037] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:53.350988 18975 raft_consensus.cc:3037] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:53.354816 16752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 83) took 1079 ms. Trace:
I20260810 18:51:53.319790 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:53.329074 19047 raft_consensus.cc:2781] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260810 18:51:53.360460 19140 log.cc:924] Time spent T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.853s user 0.006s sys 0.000s
I20260810 18:51:53.361791 16504 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56468 (request call id 81) took 1071 ms. Trace:
I20260810 18:51:53.288879 18656 consensus_queue.cc:1046] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260810 18:51:53.332127 18989 raft_consensus.cc:3032] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:51:53.328850 18870 raft_consensus.cc:2726] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260810 18:51:53.371201 17786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55586 (request call id 221) took 2082 ms. Trace:
I20260810 18:51:53.399907 16745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1312) took 1281 ms. Trace:
I20260810 18:51:53.331452 19129 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:53.328095 19077 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:53.309052 19068 consensus_queue.cc:1046] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
I20260810 18:51:53.405838 16745 rpcz_store.cc:276] 0810 18:51:52.118401 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:52.123160 (+ 4759us) service_pool.cc:227] Handling call
0810 18:51:52.123537 (+ 377us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:52.181303 (+ 57766us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:52.181365 (+ 62us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:52.181418 (+ 53us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:51:52.181471 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:53.399658 (+1218187us) spinlock_profiling.cc:243] Waited 57.5 ms on lock 0x7b540029ca54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:51:53.399710 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:53.399848 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":132300672}
I20260810 18:51:53.314738 17783 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "649d43610ba24875a4418e39b25126ba"
dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
from {username='slave'} at 127.0.0.1:55586
I20260810 18:51:53.407316 19068 raft_consensus.cc:3032] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:51:53.407936 19068 raft_consensus.cc:729] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:53.408021 17783 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:51:53.408697 17783 raft_consensus.cc:3037] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:53.409562 19068 consensus_queue.cc:260] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:53.386788 16752 rpcz_store.cc:276] 0810 18:51:52.275693 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:52.286763 (+ 11070us) service_pool.cc:227] Handling call
0810 18:51:52.701920 (+415157us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:52.702128 (+ 208us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:52.702156 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:52.702195 (+ 39us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:51:52.702221 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:52.702276 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:53.354633 (+652357us) spinlock_profiling.cc:243] Waited 415 ms on lock 0x7b54002391d0. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:53.354750 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":953871616}
I20260810 18:51:53.406281 19129 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:53.395942 16504 rpcz_store.cc:276] 0810 18:51:52.290570 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:53.100165 (+809595us) service_pool.cc:227] Handling call
0810 18:51:53.233658 (+133493us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:53.255685 (+ 22027us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:53.361588 (+105903us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b54001e14d0. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:53.361729 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":254220672}
I20260810 18:51:53.399875 18989 raft_consensus.cc:729] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:53.382093 19164 raft_consensus.cc:3037] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
W20260810 18:51:53.403229 16450 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.618s user 0.033s sys 0.055s
W20260810 18:51:53.439850 16450 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.654s user 0.034s sys 0.055s
I20260810 18:51:53.405536 17786 rpcz_store.cc:276] 0810 18:51:51.288667 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:51.327491 (+ 38824us) service_pool.cc:227] Handling call
0810 18:51:52.312058 (+984567us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:52.341595 (+ 29537us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:53.370922 (+1029327us) spinlock_profiling.cc:243] Waited 984 ms on lock 0x7b54004f06d0. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:53.371119 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2263213824}
I20260810 18:51:53.251214 18949 consensus_queue.cc:260] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:53.383734 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:53.447649 16504 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa"
dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
from {username='slave'} at 127.0.0.1:52338
I20260810 18:51:53.448335 16504 raft_consensus.cc:482] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:51:53.448755 16504 raft_consensus.cc:3037] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:53.412002 17775 raft_consensus.cc:1261] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260810 18:51:53.387840 16575 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.601s user 0.015s sys 0.058s
W20260810 18:51:53.449420 16575 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.663s user 0.015s sys 0.058s
W20260810 18:51:53.449838 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 663630us
I20260810 18:51:53.452977 16636 raft_consensus.cc:1226] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 2. Current term is 3. Ops: []
I20260810 18:51:53.393482 19047 raft_consensus.cc:482] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
W20260810 18:51:53.394968 18687 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.015s sys 0.032s
W20260810 18:51:53.430615 19125 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.019s sys 0.033s
I20260810 18:51:53.467216 16633 raft_consensus.cc:1226] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: []
W20260810 18:51:53.440124 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 662314us
W20260810 18:51:53.443341 16693 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.013s sys 0.066s
W20260810 18:51:53.415295 19111 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.600s user 0.023s sys 0.040s
I20260810 18:51:53.456566 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:53.454226 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:53.461450 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:53.288457 19168 consensus_queue.cc:260] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:51:53.458634 18827 consensus_meta.cc:287] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.617s user 0.005s sys 0.009s
W20260810 18:51:53.474378 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 52116.5us
I20260810 18:51:53.463766 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:51:53.468406 16693 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.607s user 0.013s sys 0.067s
I20260810 18:51:53.418272 19068 raft_consensus.cc:3037] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:53.474553 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:53.475184 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:53.448655 18946 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:51:53.467433 19125 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.618s user 0.020s sys 0.034s
I20260810 18:51:53.464402 19047 raft_consensus.cc:3037] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:53.486616 16750 raft_consensus.cc:2396] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260810 18:51:53.487425 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:53.352073 19112 consensus_queue.cc:260] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:53.448415 18981 consensus_queue.cc:1046] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20260810 18:51:53.500159 16450 leader_election.cc:302] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:51:53.466719 18728 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.612s user 0.004s sys 0.050s
I20260810 18:51:53.436739 19151 consensus_queue.cc:1046] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20260810 18:51:53.492774 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:53.416952 18709 consensus_queue.cc:260] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:53.491901 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:53.466270 18687 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.605s user 0.015s sys 0.032s
I20260810 18:51:53.497473 18989 consensus_queue.cc:260] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:53.504272 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:53.510852 16624 raft_consensus.cc:2379] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 4: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:53.485656 18862 leader_election.cc:288] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:53.485078 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:53.508291 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:51:53.473620 19111 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.655s user 0.024s sys 0.041s
I20260810 18:51:53.436925 19147 consensus_queue.cc:1046] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260810 18:51:53.491885 16495 raft_consensus.cc:1226] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
W20260810 18:51:53.519598 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 231973us
W20260810 18:51:53.519614 16702 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.575s user 0.000s sys 0.041s
W20260810 18:51:53.520809 16702 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.706s user 0.001s sys 0.041s
I20260810 18:51:53.520771 16573 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:53.479123 19169 consensus_queue.cc:1046] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
W20260810 18:51:53.506829 18728 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.652s user 0.004s sys 0.050s
I20260810 18:51:53.514006 16623 raft_consensus.cc:3037] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:53.510181 16450 leader_election.cc:302] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:53.523502 16504 raft_consensus.cc:504] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:53.507195 18808 raft_consensus.cc:3032] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:51:53.507493 19112 raft_consensus.cc:3037] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:53.527354 16504 leader_election.cc:288] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:53.522815 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:53.530634 18942 consensus_queue.cc:1035] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:51:53.515928 19174 consensus_queue.cc:1035] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.116s
I20260810 18:51:53.532090 16500 raft_consensus.cc:2454] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
I20260810 18:51:53.522778 18946 raft_consensus.cc:2726] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:51:53.510954 19129 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:53.496081 19152 consensus_queue.cc:1046] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20260810 18:51:53.533820 16573 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
W20260810 18:51:53.494082 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 315579us
I20260810 18:51:53.525490 18989 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:53.525144 18808 raft_consensus.cc:729] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:53.545769 16504 raft_consensus.cc:1226] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:53.534075 18709 raft_consensus.cc:3037] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:53.545962 16623 raft_consensus.cc:2454] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:51:53.534260 19094 leader_election.cc:288] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:53.566087 16504 raft_consensus.cc:1226] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:53.513674 19154 consensus_queue.cc:1035] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260810 18:51:53.599996 16504 raft_consensus.cc:1226] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 2. Current term is 3. Ops: []
W20260810 18:51:53.507339 18870 consensus_queue.cc:1159] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "831925dd8b644ebab7eb0d672a94eddd" responder_term: 1 status { last_received { term: 1 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 4 } }
I20260810 18:51:53.591472 16623 raft_consensus.cc:1226] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:53.545251 19177 raft_consensus.cc:3032] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:53.605113 19177 raft_consensus.cc:729] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
W20260810 18:51:53.593127 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 45969.8us
I20260810 18:51:53.606487 19177 consensus_queue.cc:260] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:53.607760 19152 consensus_queue.cc:1046] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:51:53.608179 19177 raft_consensus.cc:3037] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:53.563553 18728 raft_consensus.cc:2781] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:51:53.592725 16624 raft_consensus.cc:1226] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 3. Current term is 4. Ops: []
I20260810 18:51:53.595253 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:53.595991 16495 raft_consensus.cc:1226] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:53.596644 18516 leader_election.cc:288] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:53.549629 18942 raft_consensus.cc:2726] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260810 18:51:53.609428 16450 leader_election.cc:302] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:53.611423 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:53.545610 16573 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:53.617458 16501 raft_consensus.cc:2454] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:53.612363 19154 raft_consensus.cc:3032] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Stepping down as leader of term 1
W20260810 18:51:53.609040 18948 consensus_queue.cc:1159] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0397d760589d40d3b92ef141cdfc2dbb" responder_term: 2 status { last_received { term: 1 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260810 18:51:53.632326 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 86818.3us
I20260810 18:51:53.624176 17786 raft_consensus.cc:1226] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 3. Ops: []
I20260810 18:51:53.624749 16621 raft_consensus.cc:3032] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:53.626262 18808 consensus_queue.cc:260] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:53.627394 19029 raft_consensus.cc:3037] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:53.631023 19190 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:53.650465 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:53.634529 16504 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:52330 (request call id 788) took 2932 ms. Trace:
I20260810 18:51:53.636890 17778 raft_consensus.cc:1226] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:53.651825 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:53.655251 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:53.656173 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:53.656955 16749 raft_consensus.cc:3037] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:53.658998 16752 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:51:53.632953 18946 raft_consensus.cc:3037] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:53.666595 16747 raft_consensus.cc:1226] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 2. Current term is 3. Ops: []
I20260810 18:51:53.651474 16504 rpcz_store.cc:276] 0810 18:51:50.697322 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:53.602059 (+2904737us) service_pool.cc:227] Handling call
0810 18:51:53.629218 (+ 27159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:51:53.667189 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:53.656219 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:53.663898 19017 consensus_queue.cc:1046] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:51:53.638110 16621 raft_consensus.cc:729] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:53.670076 16752 raft_consensus.cc:2379] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 2: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:51:53.672683 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:53.634032 19154 raft_consensus.cc:729] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:53.680665 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:53.681167 19129 raft_consensus.cc:3032] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:51:53.682709 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:53.679406 16749 raft_consensus.cc:2454] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:53.684335 16742 raft_consensus.cc:1226] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:53.685256 18843 consensus_queue.cc:1046] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:53.685770 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:53.661229 18942 consensus_queue.cc:1046] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260810 18:51:53.688004 18730 raft_consensus.cc:3032] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:53.687947 19163 raft_consensus.cc:482] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:53.689111 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:53.650907 18728 raft_consensus.cc:686] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:53.691772 16504 raft_consensus.cc:2379] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 2: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:53.638197 18862 raft_consensus.cc:3037] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:53.668733 19190 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:53.703553 18941 consensus_meta.cc:287] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.546s user 0.013s sys 0.000s
I20260810 18:51:53.703401 19169 consensus_queue.cc:1046] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260810 18:51:53.704221 19099 consensus_meta.cc:287] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.501s user 0.011s sys 0.002s
I20260810 18:51:53.705103 19162 raft_consensus.cc:482] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:53.705643 18631 raft_consensus.cc:482] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:53.666563 18808 raft_consensus.cc:3037] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:53.705906 19154 consensus_queue.cc:260] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:53.708186 18931 raft_consensus.cc:3032] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:53.710337 19077 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:53.710574 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:53.714577 19111 raft_consensus.cc:3032] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:51:53.650321 18516 consensus_queue.cc:1046] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20260810 18:51:53.661917 19181 raft_consensus.cc:482] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:53.721133 16632 raft_consensus.cc:3037] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:53.721283 18951 raft_consensus.cc:3032] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:53.675439 17780 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55570 (request call id 334) took 1242 ms. Trace:
I20260810 18:51:53.788595 19154 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:53.794762 17780 rpcz_store.cc:276] 0810 18:51:52.426982 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:52.429918 (+ 2936us) service_pool.cc:227] Handling call
0810 18:51:52.449763 (+ 19845us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:52.467223 (+ 17460us) raft_consensus.cc:1520] Early marking committed up to index 4
0810 18:51:53.247433 (+780210us) log.cc:842] Serialized 14 byte log entry
0810 18:51:53.634469 (+387036us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:53.639941 (+ 5472us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:51:53.647670 (+ 7729us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:53.654595 (+ 6925us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:53.669701 (+ 15106us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1129,"raft.queue_time_us":25607,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":153,"thread_start_us":318391,"threads_started":1,"wal-append.queue_time_us":452217}
I20260810 18:51:53.732743 16450 leader_election.cc:302] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:53.798225 18951 raft_consensus.cc:729] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
W20260810 18:51:53.801497 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 68860.6us
I20260810 18:51:53.741319 18970 leader_election.cc:288] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:53.662109 19097 consensus_queue.cc:1046] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20260810 18:51:53.698063 19129 raft_consensus.cc:729] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:53.734222 16752 raft_consensus.cc:1261] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260810 18:51:53.722245 18730 raft_consensus.cc:729] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:53.737576 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:53.709625 17783 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:53.755131 17778 raft_consensus.cc:1226] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:53.766933 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:53.832404 16504 raft_consensus.cc:2379] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 2: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:53.769236 16575 leader_election.cc:302] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:53.756632 18809 raft_consensus.cc:3032] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:53.746461 16502 raft_consensus.cc:2454] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
I20260810 18:51:53.772853 19162 raft_consensus.cc:504] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:53.727747 19163 raft_consensus.cc:504] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:53.778436 18631 raft_consensus.cc:504] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:53.770112 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:53.781383 19190 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:53.858474 18631 leader_election.cc:288] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:53.781226 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:53.784292 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:53.785795 19047 raft_consensus.cc:504] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:53.860253 16508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1293) took 1858 ms. Trace:
I20260810 18:51:53.860723 16508 rpcz_store.cc:276] 0810 18:51:52.002041 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:52.180439 (+178398us) service_pool.cc:227] Handling call
0810 18:51:53.002590 (+822151us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:53.086664 (+ 84074us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:53.860069 (+773405us) spinlock_profiling.cc:243] Waited 822 ms on lock 0x7b5400284e50. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:53.860196 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1890124032}
I20260810 18:51:53.861429 19047 leader_election.cc:288] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:53.782321 18931 raft_consensus.cc:729] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:53.832640 18580 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:53.863253 19158 raft_consensus.cc:3037] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:53.862911 18580 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:53.863010 18931 consensus_queue.cc:260] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:53.864851 18931 raft_consensus.cc:3037] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:53.786034 19077 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:53.865703 18580 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:53.789572 18728 consensus_queue.cc:237] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:53.784014 19111 raft_consensus.cc:729] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:53.808555 16450 leader_election.cc:302] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:53.791198 19181 raft_consensus.cc:504] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:53.788909 19156 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:53.873261 19156 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:53.873704 16746 consensus_queue.cc:260] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:53.875300 16746 raft_consensus.cc:3037] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 3
I20260810 18:51:53.876150 19156 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:53.838944 19125 raft_consensus.cc:2726] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:51:53.879012 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:53.879773 16496 consensus_meta.cc:287] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.633s user 0.010s sys 0.003s
I20260810 18:51:53.879971 16631 raft_consensus.cc:2379] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 2: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:53.880174 16496 raft_consensus.cc:2454] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
W20260810 18:51:53.838871 19138 log.cc:924] Time spent T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.082s user 0.001s sys 0.000s
I20260810 18:51:53.880930 16622 raft_consensus.cc:2454] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 2.
I20260810 18:51:53.881392 16631 raft_consensus.cc:1226] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:53.881699 16632 raft_consensus.cc:2454] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 2.
W20260810 18:51:53.834707 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 65592.7us
I20260810 18:51:53.882501 16623 raft_consensus.cc:1226] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:53.882666 16575 leader_election.cc:302] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:53.833254 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:53.807691 18852 raft_consensus.cc:2726] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260810 18:51:53.840790 19192 log.cc:924] Time spent T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.097s user 0.000s sys 0.001s
I20260810 18:51:53.843027 19162 leader_election.cc:288] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:53.835745 18809 raft_consensus.cc:729] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:53.846329 18730 consensus_queue.cc:260] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:53.852126 18946 raft_consensus.cc:2726] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:51:53.852331 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:53.854838 16501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52338 (request call id 1253) took 1009 ms. Trace:
I20260810 18:51:53.858891 19141 raft_consensus.cc:3032] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:51:53.871793 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:53.868083 19111 consensus_queue.cc:260] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:53.830269 19105 raft_consensus.cc:3032] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:51:53.883610 16504 raft_consensus.cc:3037] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
W20260810 18:51:53.831825 18883 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.332s user 0.030s sys 0.030s
I20260810 18:51:53.810674 16743 raft_consensus.cc:2454] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:53.885354 17786 raft_consensus.cc:1226] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:53.886348 16575 leader_election.cc:302] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
W20260810 18:51:53.891254 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 82824.3us
I20260810 18:51:53.799389 18951 consensus_queue.cc:260] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:53.922811 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:53.908159 16448 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:54.006278 16450 leader_election.cc:302] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:54.008920 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:54.009078 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 122796us
I20260810 18:51:54.009203 16573 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:54.009596 16626 raft_consensus.cc:2454] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:54.010524 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:54.011288 16626 raft_consensus.cc:2379] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 2: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:53.899231 16631 raft_consensus.cc:1226] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:53.896940 16746 raft_consensus.cc:2454] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 3.
I20260810 18:51:54.014711 17785 raft_consensus.cc:2379] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 2: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:51:53.880867 19181 leader_election.cc:288] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:54.024891 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:54.025740 17778 raft_consensus.cc:2379] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 4: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:51:54.026178 16575 leader_election.cc:302] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:53.898075 17727 leader_election.cc:302] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:51:54.027405 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 136853us
I20260810 18:51:54.029199 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:54.029639 19145 raft_consensus.cc:2726] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260810 18:51:54.030146 19145 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:53.890219 18709 raft_consensus.cc:2726] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:51:54.015220 16502 raft_consensus.cc:1226] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 2. Current term is 3. Ops: []
I20260810 18:51:54.018566 19077 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:51:54.018826 19172 consensus_queue.cc:1159] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "831925dd8b644ebab7eb0d672a94eddd" responder_term: 2 status { last_received { term: 1 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260810 18:51:54.037482 19197 raft_consensus.cc:2726] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260810 18:51:54.019294 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 111257us
I20260810 18:51:54.019913 19199 raft_consensus.cc:3037] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:54.021126 18989 raft_consensus.cc:3037] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:53.968448 19163 leader_election.cc:288] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:54.022321 19156 raft_consensus.cc:2726] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:51:54.035638 19200 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:54.022569 16633 raft_consensus.cc:2454] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
W20260810 18:51:54.023182 18903 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
W20260810 18:51:54.023327 19193 log.cc:924] Time spent T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.200s user 0.001s sys 0.000s
I20260810 18:51:54.024075 16450 leader_election.cc:302] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:53.885661 19017 raft_consensus.cc:2726] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:54.006917 16501 rpcz_store.cc:276] 0810 18:51:52.845225 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:53.088179 (+242954us) service_pool.cc:227] Handling call
0810 18:51:53.854613 (+766434us) spinlock_profiling.cc:243] Waited 451 ms on lock 0x7b5400285854. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:53.854774 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1036202880}
W20260810 18:51:54.030987 16573 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.14.32.193:40791 timed out after 1.560s (SENT) [suppressed 14 similar messages]
I20260810 18:51:53.897415 18982 raft_consensus.cc:2781] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260810 18:51:54.013119 18883 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.514s user 0.032s sys 0.033s
I20260810 18:51:54.063956 16450 leader_election.cc:302] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:54.039675 16575 leader_election.cc:302] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
W20260810 18:51:54.042438 16746 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37716 (request call id 78) took 2343 ms (client timeout 1502 ms). Trace:
W20260810 18:51:54.065095 16746 rpcz_store.cc:269] 0810 18:51:51.670996 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:51.671604 (+ 608us) service_pool.cc:227] Handling call
0810 18:51:52.941181 (+1269577us) spinlock_profiling.cc:243] Waited 214 ms on lock 0x7b540022ac20. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae808a730 00007f1ae80fdeff 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86
0810 18:51:53.873303 (+932122us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x7b540022ac20. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae8090951 00007f1ae80f1321 00007f1ae80f2b4e 00007f1ae80fe228 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2
0810 18:51:54.014669 (+141366us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":894152832}
I20260810 18:51:54.041034 16624 raft_consensus.cc:1226] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:54.034083 16504 raft_consensus.cc:2454] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
I20260810 18:51:54.068256 16504 raft_consensus.cc:1226] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:54.005028 19105 raft_consensus.cc:729] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:51:54.044483 16448 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:54.034687 16626 raft_consensus.cc:1226] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:54.012908 19141 raft_consensus.cc:729] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:51:54.049989 16631 raft_consensus.cc:1226] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:54.073836 16626 raft_consensus.cc:1226] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:54.072482 16504 raft_consensus.cc:1226] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260810 18:51:54.043435 18975 consensus_meta.cc:287] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.657s user 0.007s sys 0.006s
I20260810 18:51:54.035429 19145 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:54.040743 18951 raft_consensus.cc:3037] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:54.031571 18730 raft_consensus.cc:3037] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:54.065129 19156 raft_consensus.cc:2781] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:54.057933 16506 raft_consensus.cc:1226] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 2. Current term is 3. Ops: []
W20260810 18:51:54.058319 16500 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52338 (request call id 1251) took 1278 ms (client timeout 1560 ms). Trace:
I20260810 18:51:53.942255 18809 consensus_queue.cc:260] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:54.030429 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:54.030541 18946 raft_consensus.cc:2726] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:54.061582 18982 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:54.088414 18982 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:54.088510 17781 raft_consensus.cc:2454] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:51:54.078292 19038 consensus_queue.cc:1046] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20260810 18:51:54.069885 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:54.074258 16450 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:54.096118 16501 raft_consensus.cc:2379] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 2: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:54.072901 18931 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:54.077001 19024 raft_consensus.cc:2726] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:54.077776 16504 raft_consensus.cc:1226] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3. Ops: []
I20260810 18:51:54.077883 16575 leader_election.cc:302] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:54.077975 18709 raft_consensus.cc:2726] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:51:54.056329 18858 raft_consensus.cc:3032] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:54.082267 16626 raft_consensus.cc:1226] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:54.078469 19077 raft_consensus.cc:2781] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:54.069887 19018 raft_consensus.cc:3032] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Stepping down as leader of term 2
W20260810 18:51:54.085616 16500 rpcz_store.cc:269] 0810 18:51:52.779540 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:52.795934 (+ 16394us) service_pool.cc:227] Handling call
0810 18:51:54.057617 (+1261683us) spinlock_profiling.cc:243] Waited 38.3 ms on lock 0x7b5400116f54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:54.058256 (+ 639us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":88090880}
I20260810 18:51:54.112267 16450 leader_election.cc:302] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:54.048018 19200 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:54.097909 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
W20260810 18:51:54.117697 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 86739.2us
I20260810 18:51:54.119544 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:54.085556 17783 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:54.106168 16575 leader_election.cc:302] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:54.108670 19145 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:54.114077 18989 raft_consensus.cc:2726] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:54.115969 19162 raft_consensus.cc:3037] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:54.115324 18982 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:54.114311 16626 raft_consensus.cc:1226] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 3. Current term is 4. Ops: []
I20260810 18:51:54.117797 16746 raft_consensus.cc:3032] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:54.096139 19105 consensus_queue.cc:260] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:54.115665 16631 raft_consensus.cc:1226] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260810 18:51:54.137228 16746 raft_consensus.cc:729] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:54.141636 16573 leader_election.cc:302] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:54.139355 16575 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:54.142333 16746 consensus_queue.cc:260] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:54.143326 16746 raft_consensus.cc:3037] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:54.144301 16750 raft_consensus.cc:3032] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:54.148718 16750 raft_consensus.cc:729] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:54.112232 18858 raft_consensus.cc:729] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:54.102319 19156 raft_consensus.cc:686] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 2 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:54.149363 16573 leader_election.cc:302] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:54.133298 19015 consensus_queue.cc:1046] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260810 18:51:54.112362 18975 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:54.150806 18516 raft_consensus.cc:3032] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:54.140300 18580 consensus_queue.cc:1035] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260810 18:51:54.112722 18872 consensus_queue.cc:1046] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20260810 18:51:54.111927 18946 raft_consensus.cc:2726] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:51:54.158922 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:54.116840 19077 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:54.121359 19018 raft_consensus.cc:729] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:51:54.160368 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:51:54.248293 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 71091.2us
I20260810 18:51:54.249120 16501 raft_consensus.cc:2454] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 2.
I20260810 18:51:54.161387 16746 raft_consensus.cc:2454] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 2.
I20260810 18:51:54.158480 18948 raft_consensus.cc:2726] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:54.162624 16500 raft_consensus.cc:1226] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:54.162447 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:54.164508 16631 raft_consensus.cc:1226] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:54.165174 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:54.155405 18989 raft_consensus.cc:2726] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 18:51:54.249953 19029 consensus_meta.cc:287] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.595s user 0.011s sys 0.000s
I20260810 18:51:54.167340 16695 leader_election.cc:302] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:54.115641 19135 consensus_queue.cc:260] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:54.168532 19200 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:54.150362 18709 raft_consensus.cc:2781] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:51:54.169446 19190 raft_consensus.cc:2726] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:51:54.171219 18982 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:54.145663 19197 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:54.161367 18862 raft_consensus.cc:2726] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:51:54.175832 16750 consensus_queue.cc:260] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:54.172555 19105 raft_consensus.cc:3037] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:54.111976 18931 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:54.174476 18870 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:54.244737 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:51:54.399467 16743 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43726 (request call id 1278) took 1531 ms (client timeout 1939 ms). Trace:
I20260810 18:51:54.399607 16626 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
W20260810 18:51:54.400019 16743 rpcz_store.cc:269] 0810 18:51:52.868045 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:52.935143 (+ 67098us) service_pool.cc:227] Handling call
0810 18:51:54.399261 (+1464118us) spinlock_profiling.cc:243] Waited 841 ms on lock 0x7b540004b0d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:54.399389 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1934228352}
I20260810 18:51:54.247776 16636 raft_consensus.cc:2454] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
I20260810 18:51:54.254487 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:54.217108 18905 raft_consensus.cc:2781] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:54.166277 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:54.241075 18516 raft_consensus.cc:729] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:54.240492 19024 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:54.315502 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:54.242295 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:51:54.317071 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 149788us
I20260810 18:51:54.248627 19077 raft_consensus.cc:3037] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:54.248791 18883 raft_consensus.cc:482] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:54.324333 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:54.326294 18946 raft_consensus.cc:482] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:54.326996 18989 consensus_queue.cc:1046] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:51:54.379428 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:54.437733 16747 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:54.245867 18827 raft_consensus.cc:482] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:54.382381 17727 leader_election.cc:302] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:54.243645 18941 consensus_queue.cc:1046] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.074s
I20260810 18:51:54.437023 18883 raft_consensus.cc:504] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:54.443536 18941 raft_consensus.cc:3032] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:54.443955 18941 raft_consensus.cc:729] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:54.242650 19068 consensus_queue.cc:1046] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.076s
I20260810 18:51:54.323266 18948 raft_consensus.cc:482] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:54.390947 16750 raft_consensus.cc:3037] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:54.250625 17724 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:54.423976 18905 raft_consensus.cc:686] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 2 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
W20260810 18:51:54.448406 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 197851us
I20260810 18:51:54.403419 16636 raft_consensus.cc:1226] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:54.378580 18656 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:54.404482 16508 raft_consensus.cc:1226] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260810 18:51:54.380815 19163 raft_consensus.cc:2781] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:54.246112 18975 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:54.419589 16450 leader_election.cc:302] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:54.384207 19200 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:54.392194 19199 raft_consensus.cc:2726] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:54.395007 18709 raft_consensus.cc:686] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:54.382365 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:51:54.393950 18808 consensus_queue.cc:1159] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d304ac47fdf846de81337057a4ef7bea" responder_term: 2 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260810 18:51:54.424513 19158 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:54.445365 18827 raft_consensus.cc:504] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:54.452491 18516 consensus_queue.cc:260] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:54.416203 18870 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:54.444646 18883 leader_election.cc:288] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:54.401902 19018 consensus_queue.cc:260] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:51:54.471230 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 51776.5us
I20260810 18:51:54.492295 16750 raft_consensus.cc:1261] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260810 18:51:54.494343 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 242813us
I20260810 18:51:54.453612 19029 raft_consensus.cc:2781] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:54.390111 18858 consensus_queue.cc:260] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:54.479792 16501 raft_consensus.cc:1226] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:54.478406 18730 raft_consensus.cc:3037] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:54.480947 19158 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:54.467255 19099 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:54.462622 18905 consensus_queue.cc:237] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:54.466190 18709 consensus_queue.cc:237] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:54.469931 16742 raft_consensus.cc:2379] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 2: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:51:54.464517 19200 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:51:54.490487 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 226424us
I20260810 18:51:54.444804 18941 consensus_queue.cc:260] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:54.485316 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:54.402112 19197 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:54.427325 19024 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:54.470451 16747 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:51:54.464628 19038 raft_consensus.cc:2781] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:54.523396 19038 raft_consensus.cc:686] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:51:54.524399 19038 consensus_queue.cc:237] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:54.525580 19168 raft_consensus.cc:3037] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:54.528508 16621 consensus_queue.cc:260] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:54.510064 19099 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:51:54.531687 19205 log.cc:924] Time spent T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.081s user 0.001s sys 0.000s
I20260810 18:51:54.533120 16624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1318) took 1020 ms. Trace:
I20260810 18:51:54.533442 16624 rpcz_store.cc:276] 0810 18:51:53.512700 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:54.067389 (+554689us) service_pool.cc:227] Handling call
0810 18:51:54.067774 (+ 385us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:51:54.068030 (+ 256us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:54.068065 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:51:54.071145 (+ 3080us) log.cc:842] Serialized 26 byte log entry
0810 18:51:54.208448 (+137303us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:51:54.208487 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:54.208584 (+ 97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:51:54.532887 (+324303us) raft_consensus.cc:1671] finished
0810 18:51:54.532911 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:54.533062 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":83932,"threads_started":1,"wal-append.queue_time_us":339469}
I20260810 18:51:54.535844 16624 raft_consensus.cc:1261] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:51:54.495786 16575 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:54.591655 19198 raft_consensus.cc:3032] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Stepping down as leader of term 1
W20260810 18:51:54.591935 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 96249.7us
I20260810 18:51:54.593454 19200 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:54.593928 19111 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:54.439903 18946 raft_consensus.cc:504] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:54.592062 19198 raft_consensus.cc:729] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:54.556465 18538 consensus_queue.cc:1035] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:54.465952 16626 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:51:54.494535 18516 raft_consensus.cc:3037] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:54.453027 19177 consensus_queue.cc:1046] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20260810 18:51:54.461802 18656 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:51:54.483400 19202 log.cc:924] Time spent T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.069s user 0.000s sys 0.001s
I20260810 18:51:54.497170 19029 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:54.516934 16636 raft_consensus.cc:2454] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:54.501652 19018 raft_consensus.cc:3037] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:54.509882 18858 raft_consensus.cc:3037] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:54.541079 16633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1295) took 1082 ms. Trace:
I20260810 18:51:54.629593 16633 rpcz_store.cc:276] 0810 18:51:53.458778 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:53.468215 (+ 9437us) service_pool.cc:227] Handling call
0810 18:51:54.540915 (+1072700us) spinlock_profiling.cc:243] Waited 286 ms on lock 0x7b54001ebc54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:54.541012 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":657333504}
W20260810 18:51:54.629973 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 182045us
I20260810 18:51:54.544018 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:54.547268 16621 raft_consensus.cc:3037] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:54.551734 18970 raft_consensus.cc:3032] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:54.551926 18931 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:54.556195 16450 leader_election.cc:302] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:51:54.640384 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 84285.2us
I20260810 18:51:54.421805 19156 consensus_queue.cc:237] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:54.541281 18870 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:51:54.569370 17781 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55602 (request call id 273) took 1484 ms (client timeout 1577 ms). Trace:
I20260810 18:51:54.625707 19029 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:54.577183 16632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1298) took 1117 ms. Trace:
I20260810 18:51:54.578351 16622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 104) took 1126 ms. Trace:
I20260810 18:51:54.578806 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:54.583196 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:54.583364 18975 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:54.583518 19190 raft_consensus.cc:3032] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:51:54.584410 18827 leader_election.cc:288] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:54.584772 16506 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56468 (request call id 89) took 2186 ms. Trace:
I20260810 18:51:54.585500 18728 raft_consensus.cc:3032] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:51:54.586228 19129 consensus_queue.cc:260] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:54.587229 18941 raft_consensus.cc:3037] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:54.588125 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:54.589537 16623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1294) took 1131 ms. Trace:
I20260810 18:51:54.590852 18949 raft_consensus.cc:3037] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:54.592022 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:54.453258 18948 raft_consensus.cc:504] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:54.605146 17726 leader_election.cc:302] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:51:54.627974 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 313818us
I20260810 18:51:54.582867 19197 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:54.633648 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:54.643456 16752 raft_consensus.cc:1261] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:51:54.611528 18872 consensus_queue.cc:1035] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260810 18:51:54.647836 16636 raft_consensus.cc:1226] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: []
I20260810 18:51:54.614089 18937 consensus_queue.cc:1046] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.067s
I20260810 18:51:54.593958 19200 raft_consensus.cc:2726] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:51:54.573122 19141 consensus_queue.cc:260] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:54.624780 19065 raft_consensus.cc:482] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:54.602655 16344 catalog_manager.cc:5495] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 2 to 3, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 3 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:51:54.631665 16501 raft_consensus.cc:2362] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3. Re-sending same reply.
I20260810 18:51:54.682672 16622 rpcz_store.cc:276] 0810 18:51:53.451954 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:53.463451 (+ 11497us) service_pool.cc:227] Handling call
0810 18:51:54.578180 (+1114729us) spinlock_profiling.cc:243] Waited 395 ms on lock 0x7b54000ce254. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:54.578272 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":907576192}
W20260810 18:51:54.612797 16575 connection.cc:232] RPC call timeout handler was delayed by -1.0379133131355047s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19390000000000002s for a response
I20260810 18:51:54.689875 16506 rpcz_store.cc:276] 0810 18:51:52.398585 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:53.106263 (+707678us) service_pool.cc:227] Handling call
0810 18:51:54.057740 (+951477us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:54.085309 (+ 27569us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:54.584582 (+499273us) spinlock_profiling.cc:243] Waited 951 ms on lock 0x7b5400116f50. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:54.584713 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2187546496}
I20260810 18:51:55.084667 16624 raft_consensus.cc:2379] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 2: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
W20260810 18:51:55.084344 16496 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52338 (request call id 1246) took 2410 ms (client timeout 1570 ms). Trace:
W20260810 18:51:55.085188 16496 rpcz_store.cc:269] 0810 18:51:52.673283 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:52.749163 (+ 75880us) service_pool.cc:227] Handling call
0810 18:51:55.083746 (+2334583us) spinlock_profiling.cc:243] Waited 412 ms on lock 0x7b54000f5d54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:55.084034 (+ 288us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":947699456}
I20260810 18:51:55.085589 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:55.087965 16502 raft_consensus.cc:1226] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:55.087796 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:55.089419 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:55.089654 16502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1302) took 1392 ms. Trace:
I20260810 18:51:55.089737 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:55.090325 16501 raft_consensus.cc:2454] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
W20260810 18:51:55.076970 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 421955us
I20260810 18:51:54.689950 16632 rpcz_store.cc:276] 0810 18:51:53.459744 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:54.576408 (+1116664us) service_pool.cc:227] Handling call
0810 18:51:54.576717 (+ 309us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:54.576879 (+ 162us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:54.576902 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:54.576941 (+ 39us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:51:54.576970 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:54.577021 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:54.577128 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:51:55.096232 16636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1330) took 1204 ms. Trace:
I20260810 18:51:55.096691 16636 rpcz_store.cc:276] 0810 18:51:53.892072 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:54.614559 (+722487us) service_pool.cc:227] Handling call
0810 18:51:54.647649 (+ 33090us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:55.095571 (+447922us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:55.095990 (+ 419us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:51:55.099646 16749 raft_consensus.cc:2454] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:54.748225 17771 raft_consensus.cc:2396] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260810 18:51:55.101070 16746 raft_consensus.cc:2454] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:51:55.100320 16748 raft_consensus.cc:1261] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:51:55.105417 18852 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.100473 16626 raft_consensus.cc:1261] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260810 18:51:55.106402 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 501184us
I20260810 18:51:55.098747 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:55.106348 18852 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.107571 16501 raft_consensus.cc:2454] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
I20260810 18:51:55.109071 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:55.087380 16623 rpcz_store.cc:276] 0810 18:51:53.458395 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:53.605124 (+146729us) service_pool.cc:227] Handling call
0810 18:51:53.882308 (+277184us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:53.882929 (+ 621us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:54.589352 (+706423us) spinlock_profiling.cc:243] Waited 277 ms on lock 0x7b54002a4250. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:54.589479 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":636607872}
I20260810 18:51:55.109965 16501 raft_consensus.cc:1799] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Handling vote request from an unknown peer 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:55.110707 18709 raft_consensus.cc:482] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:51:55.111153 18852 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:55.111270 16631 raft_consensus.cc:1226] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 LEADER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3. Ops: []
I20260810 18:51:55.111850 18852 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.111673 18709 raft_consensus.cc:504] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.112766 18852 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.114744 18921 raft_consensus.cc:482] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:54.757993 16745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1338) took 1097 ms. Trace:
W20260810 18:51:54.689587 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 59379.3us
I20260810 18:51:54.644644 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:54.747370 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:54.782186 19024 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:55.071511 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:51:55.116940 16626 raft_consensus.cc:1226] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:55.090232 16502 rpcz_store.cc:276] 0810 18:51:53.697589 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:54.071417 (+373828us) service_pool.cc:227] Handling call
0810 18:51:55.087310 (+1015893us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:55.088517 (+ 1207us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:55.089438 (+ 921us) spinlock_profiling.cc:243] Waited 978 ms on lock 0x7b54000f5d50. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:55.089590 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2248540032}
I20260810 18:51:55.129458 18975 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.127821 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:55.126922 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:55.117563 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:55.126598 16506 raft_consensus.cc:2379] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 2: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:55.110680 16501 raft_consensus.cc:2396] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260810 18:51:55.119153 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:55.082201 16500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1311) took 1360 ms. Trace:
W20260810 18:51:55.128857 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 526629us
I20260810 18:51:55.118131 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:55.125324 16631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1311) took 1358 ms. Trace:
I20260810 18:51:55.118417 18905 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:51:55.116021 16745 rpcz_store.cc:276] 0810 18:51:53.661390 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:53.668284 (+ 6894us) service_pool.cc:227] Handling call
0810 18:51:53.668704 (+ 420us) raft_consensus.cc:1457] Updating replica for 2 ops
0810 18:51:53.669158 (+ 454us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:53.669187 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0810 18:51:53.679204 (+ 10017us) log.cc:842] Serialized 52 byte log entry
0810 18:51:54.059205 (+380001us) spinlock_profiling.cc:243] Waited 13.1 ms on lock 0x7b18003e39e0. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1aee0718b1 00007f1aee07e661 00007f1ae51b4345 00007f1aedfeba39 00007f1ae51d18bb 00007f1ae51cfcbf 00007f1ae51d1e22 00007f1ae7d7b5bb 00007f1ae7d81063 00007f1ae7d81411 00007f1ae80ce5cf
0810 18:51:54.134970 (+ 75765us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:51:54.382994 (+248024us) log.cc:842] Serialized 14 byte log entry
0810 18:51:54.383641 (+ 647us) log.cc:842] Serialized 14 byte log entry
0810 18:51:54.384567 (+ 926us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:54.384659 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:51:54.757690 (+373031us) raft_consensus.cc:1671] finished
0810 18:51:54.757719 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:54.757916 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11785,"prepare.queue_time_us":410469,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":79,"raft.queue_time_us":7087,"raft.run_cpu_time_us":413,"raft.run_wall_time_us":13521,"spinlock_wait_cycles":30100352,"thread_start_us":279352,"threads_started":2,"wal-append.queue_time_us":564016}
I20260810 18:51:55.229295 19065 raft_consensus.cc:504] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.230999 19129 raft_consensus.cc:3037] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:54.658648 19172 raft_consensus.cc:2781] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:51:55.233098 19105 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.124909 17783 raft_consensus.cc:2454] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
I20260810 18:51:55.225513 16344 catalog_manager.cc:5495] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260810 18:51:54.658239 17781 rpcz_store.cc:269] 0810 18:51:53.084848 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:53.098254 (+ 13406us) service_pool.cc:227] Handling call
0810 18:51:54.569159 (+1470905us) spinlock_profiling.cc:243] Waited 899 ms on lock 0x7b54004a24d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:54.569292 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2067376896}
I20260810 18:51:55.235057 19223 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.236480 16507 raft_consensus.cc:2454] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 2.
I20260810 18:51:55.115046 18709 leader_election.cc:288] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:51:54.980104 19162 consensus_meta.cc:287] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.839s user 0.011s sys 0.004s
I20260810 18:51:54.878242 19163 raft_consensus.cc:686] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:54.652164 19198 consensus_queue.cc:260] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:54.668850 19099 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:55.239010 18809 raft_consensus.cc:3037] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:54.685222 18946 leader_election.cc:288] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:51:54.878048 19236 log.cc:924] Time spent T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.076s user 0.002s sys 0.000s
I20260810 18:51:55.240502 18957 raft_consensus.cc:3037] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:55.243265 19038 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:51:55.244612 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 602854us
I20260810 18:51:55.248096 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:55.249338 19135 raft_consensus.cc:3037] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:55.250371 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:55.251868 18631 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.252741 18808 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.254908 18931 raft_consensus.cc:482] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:55.255792 18903 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.255421 18975 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.258076 19229 raft_consensus.cc:3032] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:55.265654 16502 raft_consensus.cc:1226] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 2. Current term is 3. Ops: []
I20260810 18:51:55.267284 19018 raft_consensus.cc:2781] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:55.330832 19018 raft_consensus.cc:686] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:51:55.331359 19065 leader_election.cc:288] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:55.332213 19038 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.332314 19018 consensus_queue.cc:237] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:51:55.333815 16507 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56468 (request call id 87) took 2963 ms (client timeout 2155 ms). Trace:
I20260810 18:51:55.334553 19038 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:51:55.334270 16507 rpcz_store.cc:269] 0810 18:51:52.392653 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:52.472807 (+ 80154us) service_pool.cc:227] Handling call
0810 18:51:53.611284 (+1138477us) spinlock_profiling.cc:243] Waited 3.17 ms on lock 0x7b5800001898. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed03c1b0 00007f1aed03ba94 00007f1ae6723c04 00007f1ae6723b92
0810 18:51:55.333556 (+1722272us) spinlock_profiling.cc:243] Waited 1.6 s on lock 0x7b54001374d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:55.333744 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3696707200}
I20260810 18:51:55.335148 18728 raft_consensus.cc:729] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:55.268843 19168 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.270391 16448 leader_election.cc:302] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:51:55.352633 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 82390.2us
I20260810 18:51:55.353914 19065 raft_consensus.cc:2781] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:55.354339 19065 raft_consensus.cc:482] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:55.354739 19065 raft_consensus.cc:3037] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:55.271448 16701 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:55.273125 19141 raft_consensus.cc:3037] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:55.273944 18656 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:55.358639 18982 consensus_queue.cc:1035] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:51:55.258118 19197 raft_consensus.cc:482] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.359889 19197 raft_consensus.cc:504] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.362600 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:55.363482 16622 raft_consensus.cc:2454] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
W20260810 18:51:55.296236 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 203880us
W20260810 18:51:55.126322 17771 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55570 (request call id 338) took 1858 ms (client timeout 1878 ms). Trace:
I20260810 18:51:55.364475 19168 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:51:54.977597 19077 consensus_meta.cc:287] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.541s user 0.005s sys 0.010s
I20260810 18:51:55.366154 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:55.366034 19077 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.367796 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:55.368263 19077 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:55.369091 19077 raft_consensus.cc:482] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.369880 19077 raft_consensus.cc:504] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.370644 16450 leader_election.cc:302] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:55.118184 18948 leader_election.cc:288] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:55.371780 18949 raft_consensus.cc:2781] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:54.688103 19190 raft_consensus.cc:729] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:55.372257 18949 raft_consensus.cc:482] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:55.372788 18949 raft_consensus.cc:3037] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
W20260810 18:51:55.268075 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 132251us
I20260810 18:51:55.373347 19190 consensus_queue.cc:260] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.374442 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:55.336266 16507 raft_consensus.cc:1226] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 2 LEADER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:54.889927 19145 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:54.903134 19029 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.323045 19105 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.339740 16505 raft_consensus.cc:1261] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260810 18:51:55.375056 19190 raft_consensus.cc:3037] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:54.877159 19158 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:55.375599 16507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1320) took 1200 ms. Trace:
I20260810 18:51:54.877875 18730 raft_consensus.cc:2726] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:55.339689 18905 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.384276 16507 rpcz_store.cc:276] 0810 18:51:54.175471 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:55.335478 (+1160007us) service_pool.cc:227] Handling call
0810 18:51:55.335981 (+ 503us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:55.375287 (+ 39306us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:55.375512 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:51:55.386300 16507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1321) took 1210 ms. Trace:
I20260810 18:51:55.386698 16507 rpcz_store.cc:276] 0810 18:51:54.175748 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:55.385253 (+1209505us) service_pool.cc:227] Handling call
0810 18:51:55.385673 (+ 420us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:55.385885 (+ 212us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:55.385913 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:55.385986 (+ 73us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:51:55.386029 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:55.386100 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:55.386211 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:51:55.115622 18921 raft_consensus.cc:504] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.403499 16622 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:51:55.391348 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:55.405355 16626 raft_consensus.cc:2454] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:55.392131 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:55.406025 18852 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:55.407413 19099 raft_consensus.cc:2726] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:51:55.393675 16448 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:55.399709 18931 raft_consensus.cc:504] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.408646 19077 leader_election.cc:288] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:55.409200 18730 raft_consensus.cc:2781] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:55.409616 18730 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:55.409942 18730 raft_consensus.cc:3037] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:55.410382 18931 leader_election.cc:288] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:55.410537 18921 leader_election.cc:288] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:55.390846 19065 raft_consensus.cc:504] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.344046 18728 consensus_queue.cc:260] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.412194 18728 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:55.340399 16502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56468 (request call id 93) took 1369 ms. Trace:
I20260810 18:51:55.413424 19038 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.413542 16502 rpcz_store.cc:276] 0810 18:51:53.971000 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:55.264557 (+1293557us) service_pool.cc:227] Handling call
0810 18:51:55.265278 (+ 721us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:55.340088 (+ 74810us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:55.340311 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:51:55.414007 19038 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.276170 16631 rpcz_store.cc:276] 0810 18:51:53.767302 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:54.389013 (+621711us) service_pool.cc:227] Handling call
0810 18:51:54.420161 (+ 31148us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:55.111885 (+691724us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:55.124996 (+ 13111us) spinlock_profiling.cc:243] Waited 674 ms on lock 0x7b5400384ad4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:51:55.125250 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1550500736}
W20260810 18:51:55.415028 19073 consensus_queue.cc:1159] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0397d760589d40d3b92ef141cdfc2dbb" responder_term: 3 status { last_received { term: 3 index: 4 } last_committed_idx: 3 last_received_current_leader { term: 3 index: 4 } }
I20260810 18:51:55.379880 19198 raft_consensus.cc:3037] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:55.416770 19038 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:55.380097 19162 raft_consensus.cc:2726] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:51:55.362716 19172 raft_consensus.cc:686] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:54.636365 18970 raft_consensus.cc:729] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:55.381150 19163 consensus_queue.cc:237] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.322997 16496 raft_consensus.cc:2454] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
I20260810 18:51:55.380219 19223 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.419363 19172 consensus_queue.cc:237] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.420239 16496 raft_consensus.cc:1226] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:55.381222 19105 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:55.399446 18631 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.226186 16500 rpcz_store.cc:276] 0810 18:51:53.722373 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:54.461222 (+738849us) service_pool.cc:227] Handling call
0810 18:51:54.461588 (+ 366us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:54.507053 (+ 45465us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:54.511533 (+ 4480us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:54.513740 (+ 2207us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:51:54.519619 (+ 5879us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:55.081349 (+561730us) spinlock_profiling.cc:243] Waited 37.4 ms on lock 0x7b5400333954. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:51:55.081462 (+ 113us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:55.081925 (+ 463us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":86120192}
I20260810 18:51:55.377701 19029 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:55.380617 17770 raft_consensus.cc:2454] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
I20260810 18:51:55.417867 18730 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.424486 18631 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:55.379176 18948 raft_consensus.cc:482] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.425488 18948 raft_consensus.cc:504] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.425951 18730 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:55.426734 16448 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:55.388528 16507 raft_consensus.cc:1226] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 2 LEADER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:55.388760 16450 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:55.389081 18982 consensus_queue.cc:1035] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260810 18:51:55.428222 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 157217us
I20260810 18:51:55.428524 16507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1271) took 1218 ms. Trace:
I20260810 18:51:55.328874 19229 raft_consensus.cc:729] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:55.428884 19162 raft_consensus.cc:2781] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:51:55.429519 19162 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
W20260810 18:51:55.364562 17771 rpcz_store.cc:269] 0810 18:51:53.267627 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:53.272972 (+ 5345us) service_pool.cc:227] Handling call
0810 18:51:55.125820 (+1852848us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b54006a04d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:55.126065 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3028535040}
I20260810 18:51:55.430038 19162 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:55.380198 16505 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1315) took 1656 ms. Trace:
I20260810 18:51:55.430073 19229 consensus_queue.cc:260] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.430380 18970 consensus_queue.cc:260] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.428943 16507 rpcz_store.cc:276] 0810 18:51:54.210222 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:55.387965 (+1177743us) service_pool.cc:227] Handling call
0810 18:51:55.388342 (+ 377us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:55.428274 (+ 39932us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:55.428459 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:51:55.431399 19229 raft_consensus.cc:3037] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:55.340965 18903 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.391707 16495 raft_consensus.cc:2379] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 3: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:55.430948 16505 rpcz_store.cc:276] 0810 18:51:53.723714 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:54.469186 (+745472us) service_pool.cc:227] Handling call
0810 18:51:55.339212 (+870026us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:55.379221 (+ 40009us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:55.379982 (+ 761us) spinlock_profiling.cc:243] Waited 867 ms on lock 0x7b54001374d0. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:55.380141 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1994319232}
I20260810 18:51:55.375815 19145 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.430109 16701 leader_election.cc:302] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:55.375391 16636 raft_consensus.cc:2379] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 2: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:55.421574 19223 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:55.342669 16344 catalog_manager.cc:5495] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: config changed from index 3 to 4, VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193) evicted. New cstate: current_term: 1 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:51:55.386601 18905 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:51:54.902170 19238 log.cc:924] Time spent T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20260810 18:51:55.427683 16448 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:55.427758 18728 raft_consensus.cc:2781] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:51:55.427623 18948 leader_election.cc:288] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:51:55.442134 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 78076.1us
I20260810 18:51:55.432140 18970 raft_consensus.cc:3037] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:55.366957 19168 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:55.446121 19244 raft_consensus.cc:2781] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:51:55.441022 18852 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.446877 16496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:52330 (request call id 789) took 1270 ms. Trace:
I20260810 18:51:55.447098 18903 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:55.395330 18808 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.448167 16495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52338 (request call id 1268) took 1430 ms. Trace:
I20260810 18:51:55.448925 16504 raft_consensus.cc:1226] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3. Ops: []
I20260810 18:51:55.454738 16740 raft_consensus.cc:2454] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:55.434799 18975 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:55.455359 16506 raft_consensus.cc:1226] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:55.497169 16506 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1269) took 1479 ms. Trace:
I20260810 18:51:55.497535 16506 rpcz_store.cc:276] 0810 18:51:54.017871 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:55.305324 (+1287453us) service_pool.cc:227] Handling call
0810 18:51:55.441324 (+136000us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:55.496881 (+ 55557us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:55.497091 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:51:55.476743 19244 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:55.498588 19244 raft_consensus.cc:3037] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:55.457065 19145 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:55.362656 19197 leader_election.cc:288] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:55.500600 18922 raft_consensus.cc:482] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.501255 18922 raft_consensus.cc:504] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.480698 16495 rpcz_store.cc:276] 0810 18:51:54.017567 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:54.030141 (+ 12574us) service_pool.cc:227] Handling call
0810 18:51:55.447881 (+1417740us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b5400117454. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:55.448090 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2963447168}
W20260810 18:51:55.477759 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 100705us
I20260810 18:51:55.444954 18728 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260810 18:51:55.458168 16621 consensus_meta.cc:287] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.617s user 0.012s sys 0.003s
I20260810 18:51:55.506742 18728 raft_consensus.cc:3037] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:51:55.506950 16621 raft_consensus.cc:2454] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
W20260810 18:51:55.507915 16621 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 1319) took 1980 ms (client timeout 1924 ms). Trace:
I20260810 18:51:55.508358 16695 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:51:55.508327 16621 rpcz_store.cc:269] 0810 18:51:53.527613 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:53.568151 (+ 40538us) service_pool.cc:227] Handling call
0810 18:51:54.528105 (+959954us) spinlock_profiling.cc:243] Waited 61.1 ms on lock 0x7b54002bbea0. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae8090951 00007f1ae80f1321 00007f1ae80f2b4e 00007f1ae80fe228 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2
0810 18:51:55.507777 (+979672us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":140524928}
I20260810 18:51:55.510053 18903 raft_consensus.cc:2781] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:55.510525 18903 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:55.510944 18903 raft_consensus.cc:3037] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:55.511528 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:55.511550 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:55.512430 16743 raft_consensus.cc:2454] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
W20260810 18:51:55.481475 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 613547us
I20260810 18:51:55.513471 18951 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.478179 16496 rpcz_store.cc:276] 0810 18:51:54.176018 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:55.421064 (+1245046us) service_pool.cc:227] Handling call
0810 18:51:55.446785 (+ 25721us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:51:55.514045 18951 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.514911 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:55.515808 17771 raft_consensus.cc:2379] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 2: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
W20260810 18:51:55.496177 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 107526us
I20260810 18:51:55.518458 16742 raft_consensus.cc:2454] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:55.410827 18949 raft_consensus.cc:504] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.512497 16748 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:51:55.483493 18852 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.518788 16450 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:55.520942 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:55.521384 18852 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:55.521713 17781 raft_consensus.cc:2454] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
I20260810 18:51:55.521981 18852 raft_consensus.cc:3037] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:55.520961 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:55.522642 18948 consensus_queue.cc:1035] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:55.523283 17770 raft_consensus.cc:2454] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:51:55.482136 18931 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.524099 18931 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.526369 18931 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:55.528048 16628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1314) took 2175 ms. Trace:
I20260810 18:51:55.528429 16628 rpcz_store.cc:276] 0810 18:51:53.351991 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:53.352538 (+ 547us) service_pool.cc:227] Handling call
0810 18:51:53.354946 (+ 2408us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:55.503581 (+2148635us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:55.503634 (+ 53us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:55.503704 (+ 70us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:51:55.503739 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:55.527643 (+ 23904us) spinlock_profiling.cc:243] Waited 2.15 s on lock 0x7b54003688d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:51:55.527683 (+ 40us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:55.527831 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":662,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":142,"spinlock_wait_cycles":4940313472}
I20260810 18:51:55.530680 16740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43738 (request call id 1352) took 1373 ms. Trace:
I20260810 18:51:55.531447 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:51:55.496274 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 66348.7us
I20260810 18:51:55.533115 18808 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:55.533679 16701 leader_election.cc:302] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:55.533941 18949 leader_election.cc:288] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:55.535941 16701 leader_election.cc:302] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:55.536589 18922 leader_election.cc:288] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:55.537386 18631 raft_consensus.cc:2781] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:51:55.531117 16740 rpcz_store.cc:276] 0810 18:51:54.157135 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:54.159358 (+ 2223us) service_pool.cc:227] Handling call
0810 18:51:55.530451 (+1371093us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b540020ba54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:55.530595 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2363660544}
I20260810 18:51:55.537914 18631 raft_consensus.cc:686] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:55.531472 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:55.539407 17786 raft_consensus.cc:2454] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:51:55.540081 16501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1318) took 1366 ms. Trace:
I20260810 18:51:55.540447 16501 rpcz_store.cc:276] 0810 18:51:54.174147 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:55.438669 (+1264522us) service_pool.cc:227] Handling call
0810 18:51:55.505212 (+ 66543us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:55.539762 (+ 34550us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:55.539791 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:55.539850 (+ 59us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:51:55.539876 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:55.539932 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:55.540028 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:51:55.542716 17724 leader_election.cc:302] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:55.545423 19250 raft_consensus.cc:2726] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:55.545818 18951 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:55.542827 16504 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1307) took 1843 ms. Trace:
I20260810 18:51:55.546418 16504 rpcz_store.cc:276] 0810 18:51:53.698969 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:54.107519 (+408550us) service_pool.cc:227] Handling call
0810 18:51:54.433250 (+325731us) spinlock_profiling.cc:243] Waited 2.46 ms on lock 0x7b5800001898. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed0502b0 00007f1aed03b7f9 00007f1aed03b2a2 00007f1ae6721684
0810 18:51:55.448617 (+1015367us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:55.482115 (+ 33498us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:55.542601 (+ 60486us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b5400117450. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:55.542749 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2340243456}
I20260810 18:51:55.547933 19099 raft_consensus.cc:2726] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:55.518404 18809 consensus_queue.cc:1046] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:51:55.548363 19130 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.548962 19130 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.550731 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:55.551204 19099 raft_consensus.cc:3032] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:55.551635 19099 raft_consensus.cc:729] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:55.546576 18951 consensus_queue.cc:1046] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260810 18:51:55.553049 18951 raft_consensus.cc:3032] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:55.553402 18951 raft_consensus.cc:729] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:55.554481 16573 leader_election.cc:302] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:51:55.551731 17771 raft_consensus.cc:2379] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 2: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
W20260810 18:51:55.555488 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 49367.3us
I20260810 18:51:55.539335 18631 consensus_queue.cc:237] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.506284 19135 raft_consensus.cc:2726] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260810 18:51:55.573151 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 97512.2us
I20260810 18:51:55.571098 16701 leader_election.cc:302] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:55.577067 16573 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:51:55.560992 19129 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.581667 18922 raft_consensus.cc:2781] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:51:55.582541 19197 raft_consensus.cc:2781] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:55.548403 18931 raft_consensus.cc:482] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.583621 19099 consensus_queue.cc:260] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.577903 18949 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.595494 16573 leader_election.cc:302] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:51:55.604039 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:55.605252 16750 raft_consensus.cc:2379] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 2: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:51:55.605522 16448 leader_election.cc:302] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:55.606492 16502 raft_consensus.cc:1226] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 2 LEADER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:55.607369 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:55.590955 18922 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:55.592164 19197 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:51:55.614825 16504 raft_consensus.cc:1226] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:55.619180 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:55.607957 19129 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.619848 17770 raft_consensus.cc:2454] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:51:55.621492 19129 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:55.622087 18905 raft_consensus.cc:2741] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260810 18:51:55.613970 18922 raft_consensus.cc:3037] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
W20260810 18:51:55.623039 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 61958.7us
I20260810 18:51:55.623466 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:55.624249 16746 raft_consensus.cc:1226] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: [3.4-3.4]
I20260810 18:51:55.624848 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:55.625612 16749 raft_consensus.cc:2454] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:55.626904 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:55.613943 18903 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.627385 18808 consensus_queue.cc:1046] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:55.627276 18931 raft_consensus.cc:504] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.628684 18808 consensus_queue.cc:1046] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:51:55.637878 16743 raft_consensus.cc:2454] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:55.636415 16695 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:55.656009 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:55.656177 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:55.656877 16752 raft_consensus.cc:2454] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 2.
I20260810 18:51:55.602370 19163 raft_consensus.cc:2726] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:55.598821 19135 raft_consensus.cc:482] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.612075 19198 raft_consensus.cc:482] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:51:55.662009 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 43551.6us
I20260810 18:51:55.654063 17726 leader_election.cc:302] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:55.611503 18957 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.630043 18809 raft_consensus.cc:3032] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:55.618031 19197 raft_consensus.cc:3037] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:55.666687 18809 raft_consensus.cc:729] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:55.666524 18957 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.639945 19099 raft_consensus.cc:3037] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:55.639544 16742 raft_consensus.cc:1261] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260810 18:51:55.629806 18808 raft_consensus.cc:3032] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:55.689102 18808 raft_consensus.cc:729] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:55.671456 18809 consensus_queue.cc:260] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.672536 19250 raft_consensus.cc:2781] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:55.673058 18957 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:55.668924 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:55.673168 18922 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.673719 18951 consensus_queue.cc:260] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.666878 19190 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.675366 19111 raft_consensus.cc:2781] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260810 18:51:55.676466 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 78987.2us
I20260810 18:51:55.676891 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:55.677597 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:55.677750 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:55.693274 16636 raft_consensus.cc:3037] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:55.678529 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:55.678632 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:55.681540 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:55.681782 16502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1272) took 1471 ms. Trace:
I20260810 18:51:55.682384 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:55.662575 19065 leader_election.cc:288] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:55.682919 19129 raft_consensus.cc:482] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.662061 19130 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:55.685039 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:55.685184 19244 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.635139 18852 raft_consensus.cc:2726] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:55.686710 18728 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.668926 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:55.687189 18870 raft_consensus.cc:2781] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:55.642031 19162 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.629655 18931 leader_election.cc:288] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:55.639668 17727 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
W20260810 18:51:55.784006 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 144599us
I20260810 18:51:55.784953 18957 raft_consensus.cc:482] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.785180 17727 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:55.785519 18957 raft_consensus.cc:504] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.785912 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:55.777225 16504 raft_consensus.cc:2454] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:51:55.633895 18903 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:55.691706 16746 raft_consensus.cc:3037] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:55.748431 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:55.617938 18949 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:51:55.759162 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 70402.7us
I20260810 18:51:55.754833 16628 raft_consensus.cc:2454] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:55.749691 16573 leader_election.cc:302] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:55.753504 16747 raft_consensus.cc:2454] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 4.
I20260810 18:51:55.764379 16631 raft_consensus.cc:2454] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:55.839329 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:55.692318 19168 raft_consensus.cc:482] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.840562 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:55.840987 19168 raft_consensus.cc:504] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.841777 19244 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:55.693526 19111 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:55.765723 17770 raft_consensus.cc:2454] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:55.781806 19162 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:55.775345 16621 raft_consensus.cc:2454] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:55.781852 18809 raft_consensus.cc:3037] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:55.789125 16508 raft_consensus.cc:1226] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 2. Current term is 4. Ops: []
I20260810 18:51:55.786571 16632 raft_consensus.cc:2454] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 3.
I20260810 18:51:55.789727 19248 raft_consensus.cc:2726] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:51:55.859615 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:55.860321 16632 raft_consensus.cc:3037] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:51:55.791322 16622 raft_consensus.cc:3037] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:55.792589 18957 leader_election.cc:288] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:55.859618 19114 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.793120 18870 raft_consensus.cc:686] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [term 2 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:55.793627 18975 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.794130 19252 raft_consensus.cc:2781] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:55.864285 18949 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:55.864465 19252 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:55.865016 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:55.865124 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:55.865926 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:55.866117 18870 consensus_queue.cc:237] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.867583 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:55.867683 17727 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:55.868237 16750 raft_consensus.cc:3037] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:51:55.868429 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:55.864858 19252 raft_consensus.cc:3037] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:55.870329 16447 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:51:55.871302 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 203756us
I20260810 18:51:55.872089 16573 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:55.872501 16747 raft_consensus.cc:1261] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260810 18:51:55.864288 18975 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.873515 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:55.873766 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:55.874080 16747 raft_consensus.cc:2454] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:55.794749 19223 consensus_queue.cc:1035] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260810 18:51:55.876058 19223 raft_consensus.cc:2781] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:51:55.876416 19223 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:55.876765 19223 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:51:55.797045 18970 raft_consensus.cc:482] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.878960 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:55.879529 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:55.756947 18951 raft_consensus.cc:3037] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:55.805117 16501 raft_consensus.cc:2379] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 5: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:55.880393 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:55.880983 16743 raft_consensus.cc:3037] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:55.812819 18808 consensus_queue.cc:260] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.794762 19141 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:55.820170 19018 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:55.883386 19141 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.883577 19018 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.884439 18957 raft_consensus.cc:2781] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:51:55.884941 18957 raft_consensus.cc:686] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:55.885407 19018 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:51:55.807215 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 46620.5us
I20260810 18:51:55.750241 16502 rpcz_store.cc:276] 0810 18:51:54.210587 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:55.414639 (+1204052us) service_pool.cc:227] Handling call
0810 18:51:55.415032 (+ 393us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:55.607113 (+192081us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:55.681514 (+ 74401us) spinlock_profiling.cc:243] Waited 178 ms on lock 0x7b54001cb6d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:51:55.681702 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":409202560}
I20260810 18:51:55.825603 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:55.886896 16633 raft_consensus.cc:2379] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 2: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:55.776455 19130 raft_consensus.cc:482] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.827205 18948 raft_consensus.cc:482] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.827704 18730 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.691246 19198 raft_consensus.cc:504] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.843001 19168 leader_election.cc:288] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:55.787969 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:51:55.803386 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 134319us
I20260810 18:51:55.691737 19135 raft_consensus.cc:504] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.693382 19190 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.776465 16624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1340) took 1186 ms. Trace:
I20260810 18:51:55.843269 19111 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:55.755640 19129 raft_consensus.cc:504] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.857509 18941 raft_consensus.cc:2726] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:51:55.861195 17726 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:51:55.871405 16508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56468 (request call id 96) took 1702 ms. Trace:
I20260810 18:51:55.879386 18970 raft_consensus.cc:504] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.919034 16508 rpcz_store.cc:276] 0810 18:51:54.169088 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:55.442267 (+1273179us) service_pool.cc:227] Handling call
0810 18:51:55.483173 (+ 40906us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:55.859146 (+375973us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:55.871157 (+ 12011us) spinlock_profiling.cc:243] Waited 306 ms on lock 0x7b5400146c54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:51:55.871344 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":703116672}
I20260810 18:51:55.819720 19029 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.883921 18949 raft_consensus.cc:2781] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:55.821770 18922 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:55.823766 18728 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:55.862416 19114 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.889127 16745 raft_consensus.cc:2454] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:55.873536 16573 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:55.894001 16750 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:51:55.894953 16636 raft_consensus.cc:2454] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:51:55.882843 18808 raft_consensus.cc:3037] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:55.874845 18975 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:55.895541 16622 raft_consensus.cc:2454] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:51:55.761536 19250 raft_consensus.cc:686] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:51:55.860401 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:55.902036 16746 raft_consensus.cc:2454] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:51:55.909756 16752 raft_consensus.cc:2454] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:51:55.910436 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:55.885421 19141 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:55.908237 16624 rpcz_store.cc:276] 0810 18:51:54.589966 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:55.090083 (+500117us) service_pool.cc:227] Handling call
0810 18:51:55.090577 (+ 494us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:55.132531 (+ 41954us) raft_consensus.cc:1520] Early marking committed up to index 2
0810 18:51:55.577301 (+444770us) log.cc:842] Serialized 14 byte log entry
0810 18:51:55.776091 (+198790us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:55.776156 (+ 65us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:51:55.776208 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:55.776280 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:55.776409 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":361358,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":734,"raft.queue_time_us":155,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":101,"thread_start_us":375997,"threads_started":2,"wal-append.queue_time_us":288392}
I20260810 18:51:55.921172 16623 raft_consensus.cc:2379] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 2: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:55.879596 16749 raft_consensus.cc:2454] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:51:55.925598 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:55.926193 16748 raft_consensus.cc:2454] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:55.929688 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:55.885928 18957 consensus_queue.cc:237] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.930371 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:55.911864 18905 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:51:55.950227 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 89134.2us
I20260810 18:51:55.952884 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:51:55.954713 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 45814us
W20260810 18:51:55.957376 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 83897.5us
I20260810 18:51:55.971001 16448 leader_election.cc:302] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:55.933566 18949 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:55.937783 18921 raft_consensus.cc:482] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:55.909641 18948 raft_consensus.cc:504] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.912190 19130 raft_consensus.cc:504] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.969585 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:55.920562 18730 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.974261 17786 raft_consensus.cc:2454] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
I20260810 18:51:55.959475 19248 raft_consensus.cc:2781] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:51:55.978235 16573 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:55.988588 16448 leader_election.cc:302] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:55.991883 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:55.994122 16622 raft_consensus.cc:1261] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260810 18:51:55.992703 16624 raft_consensus.cc:2454] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 2.
I20260810 18:51:55.996320 16501 raft_consensus.cc:2454] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 3.
I20260810 18:51:55.978916 18949 raft_consensus.cc:3037] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:55.991384 18921 raft_consensus.cc:504] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:55.948208 19029 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.978225 18922 raft_consensus.cc:2781] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:51:56.004616 16504 raft_consensus.cc:1261] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:51:55.998993 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:56.004285 16506 raft_consensus.cc:1261] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260810 18:51:55.978760 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:56.011819 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:56.012398 18921 leader_election.cc:288] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:55.976462 17770 raft_consensus.cc:2454] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:51:56.012975 19156 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:56.013574 19156 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:56.014424 19130 leader_election.cc:288] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:56.004568 18982 raft_consensus.cc:2781] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:56.015434 18982 raft_consensus.cc:482] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:56.015602 19156 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:56.015785 18982 raft_consensus.cc:3037] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:56.015969 18975 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:56.016067 18942 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:56.016721 18942 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:56.018409 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:56.019040 16505 raft_consensus.cc:2454] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 4.
I20260810 18:51:56.019379 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:56.019918 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:56.020866 19252 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:56.022190 18941 raft_consensus.cc:482] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:56.022694 18941 raft_consensus.cc:504] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:56.024685 18941 leader_election.cc:288] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:56.025156 19223 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.987860 19172 raft_consensus.cc:2781] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:56.027863 19172 raft_consensus.cc:482] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:56.028270 19172 raft_consensus.cc:3037] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:56.028412 19029 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:56.029212 18970 leader_election.cc:288] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:56.002085 18905 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:55.995779 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:56.031566 16500 raft_consensus.cc:2454] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 2.
I20260810 18:51:56.002419 16495 raft_consensus.cc:2354] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 3. Current term is 4.
I20260810 18:51:55.991444 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:56.032506 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:56.002517 19129 leader_election.cc:288] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:56.033176 16500 raft_consensus.cc:2454] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 2.
I20260810 18:51:56.035043 16701 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:56.037201 16701 leader_election.cc:302] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:56.037611 17781 raft_consensus.cc:2354] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3.
I20260810 18:51:56.006783 19145 consensus_queue.cc:1035] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260810 18:51:56.031737 18905 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:56.018749 18942 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:55.991613 19248 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:56.040268 16508 raft_consensus.cc:2454] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:51:56.008132 17726 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:51:56.012552 16506 raft_consensus.cc:2379] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 2: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:56.041311 16505 raft_consensus.cc:2454] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:51:56.041294 16447 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:56.016250 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:55.979619 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:56.042595 16632 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:51:56.043857 16450 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:56.016010 18948 leader_election.cc:288] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:56.047019 19223 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:56.080317 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:56.044993 19229 raft_consensus.cc:2741] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20260810 18:51:56.078974 19248 raft_consensus.cc:3037] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:56.079334 19029 consensus_queue.cc:1035] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260810 18:51:56.083323 19250 consensus_queue.cc:237] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:56.087563 19190 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:56.038951 19114 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:51:56.091387 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 54351.9us
I20260810 18:51:56.092288 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:56.092614 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:56.092928 17771 raft_consensus.cc:3037] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:56.093257 17770 raft_consensus.cc:2454] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:51:56.005095 18922 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:56.094062 18922 raft_consensus.cc:3037] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
W20260810 18:51:56.095340 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 48688.2us
W20260810 18:51:56.099337 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 54111.8us
I20260810 18:51:56.100319 17771 raft_consensus.cc:2454] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:51:56.044970 18931 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:56.104362 18931 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:56.091632 19248 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:56.032793 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:56.106282 16495 raft_consensus.cc:2454] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:51:56.106462 18931 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:56.107618 19248 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:56.109344 16496 raft_consensus.cc:2454] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 2.
I20260810 18:51:56.111498 19135 leader_election.cc:288] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:56.103742 19111 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:56.113765 19111 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:56.115129 18975 raft_consensus.cc:2726] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:56.111274 19172 raft_consensus.cc:504] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:56.117292 19172 leader_election.cc:288] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:51:56.118366 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 77162.7us
I20260810 18:51:56.119428 19198 leader_election.cc:288] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:56.119702 19172 raft_consensus.cc:2781] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:56.120169 19172 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:56.120587 19172 raft_consensus.cc:3037] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:56.121124 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:56.121944 17783 raft_consensus.cc:2354] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for earlier term 4. Current term is 5.
I20260810 18:51:56.122606 16701 leader_election.cc:302] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:56.122817 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:56.046821 19130 raft_consensus.cc:2781] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:56.123871 16701 leader_election.cc:302] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:56.124279 19130 raft_consensus.cc:686] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:56.125031 19163 consensus_queue.cc:1046] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260810 18:51:56.125229 19130 consensus_queue.cc:237] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:56.125631 19229 raft_consensus.cc:2726] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:56.126322 19229 raft_consensus.cc:2726] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:56.126646 19163 raft_consensus.cc:3032] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:51:56.127117 19163 raft_consensus.cc:729] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:56.127332 16701 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:56.044757 19105 consensus_queue.cc:1035] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.083s
I20260810 18:51:56.128664 16701 leader_election.cc:302] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:56.106379 17781 raft_consensus.cc:2354] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4.
I20260810 18:51:56.131196 17724 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:56.131814 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:56.134243 16447 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:56.138060 16506 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:51:56.139019 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:56.139762 16743 raft_consensus.cc:2454] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:51:56.141628 17724 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:56.128094 19163 consensus_queue.cc:260] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:51:56.142693 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 45232.3us
I20260810 18:51:56.147666 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:56.148375 16504 raft_consensus.cc:3037] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
W20260810 18:51:56.148753 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 48169.5us
I20260810 18:51:56.022991 19252 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:56.151389 16507 raft_consensus.cc:2454] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
I20260810 18:51:56.158396 18827 raft_consensus.cc:2726] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:51:56.160046 16447 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:56.148360 19114 raft_consensus.cc:2781] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260810 18:51:56.142236 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 134222us
I20260810 18:51:56.149080 19248 raft_consensus.cc:2781] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:56.148702 19198 raft_consensus.cc:482] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:56.153272 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:56.161427 18948 raft_consensus.cc:2781] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:56.159054 19029 raft_consensus.cc:2781] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:56.139364 19197 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:56.163012 19156 raft_consensus.cc:2781] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:51:56.197162 19135 raft_consensus.cc:3037] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:56.199215 19129 raft_consensus.cc:2781] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election won for term 4
W20260810 18:51:56.199421 18631 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.615s user 0.014s sys 0.064s
I20260810 18:51:56.236836 16447 leader_election.cc:398] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Vote denied by peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387) with higher term. Message: Invalid argument: T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for earlier term 4. Current term is 5.
I20260810 18:51:56.200448 19248 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:56.165295 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:56.241417 16695 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:56.241716 16628 raft_consensus.cc:2454] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 3.
I20260810 18:51:56.243089 16447 leader_election.cc:401] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260810 18:51:56.209525 18948 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
W20260810 18:51:56.257110 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 257882us
W20260810 18:51:56.266335 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 55565.1us
W20260810 18:51:56.248107 18631 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.664s user 0.015s sys 0.064s
I20260810 18:51:56.241667 19190 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:56.194692 19114 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:56.161234 19111 consensus_queue.cc:1035] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.078s
I20260810 18:51:56.211536 19163 raft_consensus.cc:3037] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
W20260810 18:51:56.245697 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 255244us
W20260810 18:51:56.272377 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 86808.6us
I20260810 18:51:56.266997 16448 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:56.239436 19029 raft_consensus.cc:686] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:51:56.240212 19156 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:56.266625 18948 raft_consensus.cc:3037] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:56.202674 19198 raft_consensus.cc:504] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:56.252104 19248 raft_consensus.cc:3037] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:56.248154 18975 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:56.282624 19114 raft_consensus.cc:3037] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:56.282914 16573 leader_election.cc:398] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Vote denied by peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791) with higher term. Message: Invalid argument: T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 3. Current term is 4.
I20260810 18:51:56.266372 18931 raft_consensus.cc:2781] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:51:56.290468 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:56.271807 19129 raft_consensus.cc:686] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:56.303478 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:56.305650 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:56.306872 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:56.289886 18730 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:56.307127 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:56.284621 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:56.293175 16448 leader_election.cc:302] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:56.272121 16496 raft_consensus.cc:2454] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 2.
I20260810 18:51:56.332295 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:56.333112 17786 raft_consensus.cc:2454] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:56.311440 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:56.243041 18827 consensus_queue.cc:1046] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260810 18:51:56.315770 16507 raft_consensus.cc:1261] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:51:56.289165 19156 raft_consensus.cc:3037] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:56.310918 18903 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:51:56.288998 16695 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:56.326584 16745 raft_consensus.cc:1226] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:56.326725 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:56.327950 16743 raft_consensus.cc:1799] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Handling vote request from an unknown peer 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:56.310141 18852 raft_consensus.cc:2781] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:51:56.342638 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:56.343180 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:56.343885 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:56.326364 18942 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:56.345144 18942 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:56.345973 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:56.349453 18949 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:56.322163 18975 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:56.351449 16501 raft_consensus.cc:2354] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for earlier term 2. Current term is 3.
I20260810 18:51:56.353006 18975 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:56.354064 16743 raft_consensus.cc:2396] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260810 18:51:56.337390 16749 raft_consensus.cc:3037] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:56.337790 18730 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:56.355719 18730 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } attrs { replace: true } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:56.357573 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:56.357280 19029 consensus_queue.cc:237] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:56.358268 18730 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:56.358433 16506 raft_consensus.cc:2454] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 5.
I20260810 18:51:56.358942 18728 raft_consensus.cc:2726] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387) with higher term. Message: Invalid argument: T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for earlier term 4. Current term is 5.
I20260810 18:51:56.323698 18957 consensus_queue.cc:1035] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260810 18:51:56.289343 19190 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:56.325523 16746 raft_consensus.cc:2454] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:56.337666 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:56.316596 18931 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:56.352936 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:56.359354 16507 raft_consensus.cc:2454] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:56.361439 16450 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:51:56.309814 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 304039us
I20260810 18:51:56.364637 19190 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:56.365615 19198 leader_election.cc:288] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:56.365460 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:56.366256 16496 raft_consensus.cc:2454] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:56.367892 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:56.368525 16507 raft_consensus.cc:2454] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:56.369029 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:51:56.369059 19273 log.cc:924] Time spent T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20260810 18:51:56.370105 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:56.370903 17778 raft_consensus.cc:2454] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
I20260810 18:51:56.371780 19099 raft_consensus.cc:2781] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:56.372170 19099 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:56.372510 19099 raft_consensus.cc:3037] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:56.372406 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:56.372706 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:56.373066 16740 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:51:56.373320 17786 raft_consensus.cc:3037] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:56.373610 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:56.374177 16748 raft_consensus.cc:3037] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:56.377777 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:56.358717 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:56.289588 18631 raft_consensus.cc:482] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:51:56.362442 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 73533.3us
I20260810 18:51:56.363534 16508 raft_consensus.cc:2379] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 3: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:56.363822 18931 raft_consensus.cc:3037] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
W20260810 18:51:56.403013 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 156609us
I20260810 18:51:56.395192 18903 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:56.397092 18922 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:56.383639 19129 consensus_queue.cc:237] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:51:56.431211 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 69868.5us
I20260810 18:51:56.435410 16573 leader_election.cc:302] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:56.435918 16750 raft_consensus.cc:2454] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:56.435998 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:56.455868 16573 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:56.445147 17786 raft_consensus.cc:2454] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:51:56.446036 17724 leader_election.cc:398] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Vote denied by peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791) with higher term. Message: Invalid argument: T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for earlier term 2. Current term is 3.
I20260810 18:51:56.447021 16695 leader_election.cc:302] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:56.457962 17724 leader_election.cc:401] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20260810 18:51:56.452095 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 159013us
I20260810 18:51:56.452371 17726 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:56.453323 16749 raft_consensus.cc:2454] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:51:56.459429 16743 raft_consensus.cc:1226] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:56.432711 16504 raft_consensus.cc:2454] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 4.
I20260810 18:51:56.456135 16746 raft_consensus.cc:2379] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 4: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:51:56.439159 16748 raft_consensus.cc:2454] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 3.
I20260810 18:51:56.461253 16450 leader_election.cc:302] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:56.460057 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:56.462397 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:56.462873 17786 raft_consensus.cc:1799] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Handling vote request from an unknown peer 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:56.463302 17786 raft_consensus.cc:2396] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for term 3 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260810 18:51:56.466159 19197 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:56.467280 19244 raft_consensus.cc:3037] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:56.468647 16740 raft_consensus.cc:2454] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 5.
I20260810 18:51:56.449373 19198 consensus_queue.cc:1035] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20260810 18:51:56.465050 18827 raft_consensus.cc:2781] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:51:56.471041 18827 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:56.471225 19279 raft_consensus.cc:2781] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:51:56.471465 18827 raft_consensus.cc:3037] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:56.471771 19279 raft_consensus.cc:686] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:56.472503 16450 leader_election.cc:302] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:56.472721 19279 consensus_queue.cc:237] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:56.479123 18631 raft_consensus.cc:504] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:56.479487 19289 raft_consensus.cc:2781] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:56.480026 19289 raft_consensus.cc:482] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:56.480449 19289 raft_consensus.cc:3037] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:56.481186 18631 leader_election.cc:288] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:56.481783 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:56.482543 17785 raft_consensus.cc:2379] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 2: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:51:56.460542 18957 raft_consensus.cc:2781] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:56.483384 18957 raft_consensus.cc:482] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:56.483714 18957 raft_consensus.cc:3037] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:56.471385 18975 raft_consensus.cc:2781] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:56.487479 18975 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:56.487917 18975 raft_consensus.cc:3037] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:56.452466 19077 raft_consensus.cc:2726] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260810 18:51:56.491916 16695 leader_election.cc:398] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Vote denied by peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387) with higher term. Message: Invalid argument: T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3.
I20260810 18:51:56.500339 16695 leader_election.cc:401] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260810 18:51:56.492643 16573 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:56.445675 18852 raft_consensus.cc:686] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
W20260810 18:51:56.500677 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 68501.4us
I20260810 18:51:56.502494 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:56.503332 16447 leader_election.cc:398] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Vote denied by peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387) with higher term. Message: Invalid argument: T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4.
I20260810 18:51:56.505494 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:56.507272 19018 raft_consensus.cc:2741] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260810 18:51:56.507833 16747 raft_consensus.cc:1226] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260810 18:51:56.507933 19018 raft_consensus.cc:3037] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Advancing to term 3
I20260810 18:51:56.492275 18949 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:56.506263 18982 raft_consensus.cc:504] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:56.510376 18903 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:56.512212 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:56.512791 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:56.514532 16747 raft_consensus.cc:2396] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260810 18:51:56.512900 16575 leader_election.cc:334] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Timed out: RequestConsensusVote RPC to 127.14.32.195:40171 timed out after 1.573s (SENT)
I20260810 18:51:56.515442 16575 leader_election.cc:302] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:56.512399 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:56.517380 17724 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:56.517851 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:56.514400 18809 raft_consensus.cc:3032] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:51:56.518288 17724 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:56.503607 17786 raft_consensus.cc:2454] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
I20260810 18:51:56.502130 16448 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:56.523452 16447 leader_election.cc:401] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260810 18:51:56.460088 19135 raft_consensus.cc:2726] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:56.529103 16573 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:51:56.531661 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:56.536090 16632 raft_consensus.cc:2454] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:51:56.531113 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:56.538442 16631 raft_consensus.cc:3037] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:56.545271 16752 raft_consensus.cc:2454] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:56.546458 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:51:56.456642 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 63762.7us
I20260810 18:51:56.548130 16633 raft_consensus.cc:1261] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:51:56.549284 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:56.478767 18948 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:56.549706 16746 raft_consensus.cc:1261] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:51:56.556581 17786 raft_consensus.cc:2354] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 3. Current term is 4.
I20260810 18:51:56.512571 19190 raft_consensus.cc:2781] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:51:56.475318 18942 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:56.560207 16747 raft_consensus.cc:1226] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 3. Current term is 4. Ops: []
I20260810 18:51:56.557776 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:56.565485 16745 raft_consensus.cc:1799] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Handling vote request from an unknown peer 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:56.565764 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:56.560783 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:56.566002 16745 raft_consensus.cc:2354] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 2. Current term is 3.
I20260810 18:51:56.526859 19198 raft_consensus.cc:2726] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:51:56.569402 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:56.524703 18809 raft_consensus.cc:729] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:56.568950 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:56.570253 16749 raft_consensus.cc:2362] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2. Re-sending same reply.
I20260810 18:51:56.572921 18982 leader_election.cc:288] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:56.576867 18948 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:56.577502 16501 raft_consensus.cc:1799] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Handling vote request from an unknown peer 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:56.577747 19244 raft_consensus.cc:2726] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:51:56.578154 16501 raft_consensus.cc:2396] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for term 3 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260810 18:51:56.578083 18931 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:56.555110 18730 raft_consensus.cc:2726] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387) with higher term. Message: Invalid argument: T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4.
I20260810 18:51:56.579500 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:56.551332 19197 raft_consensus.cc:2781] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:56.580504 19197 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:56.580878 19197 raft_consensus.cc:3037] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:56.546787 19250 raft_consensus.cc:2781] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:51:56.583278 19250 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:56.583652 19250 raft_consensus.cc:3037] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Advancing to term 4
W20260810 18:51:56.588311 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 88602.7us
I20260810 18:51:56.538164 18870 raft_consensus.cc:2781] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:51:56.578457 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:56.590487 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:56.591712 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:56.593099 17724 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:56.647392 16506 raft_consensus.cc:2454] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:56.647559 16623 raft_consensus.cc:2454] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:51:56.548898 19105 raft_consensus.cc:2781] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:51:56.604553 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:56.652226 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:56.556099 18922 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:56.654778 16506 raft_consensus.cc:3037] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:56.653582 16621 raft_consensus.cc:2454] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 3.
I20260810 18:51:56.580901 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:56.654105 16504 raft_consensus.cc:1226] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 3. Current term is 4. Ops: []
I20260810 18:51:56.662119 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:56.560077 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:56.549926 19130 raft_consensus.cc:2726] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387) with higher term. Message: Invalid argument: T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3.
W20260810 18:51:56.655436 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 181261us
I20260810 18:51:56.560582 19190 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:56.658800 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:56.677881 16507 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:51:56.659905 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:56.660714 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:56.594779 16740 raft_consensus.cc:1226] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:56.674175 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:56.683719 16636 raft_consensus.cc:3037] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:56.674688 16506 raft_consensus.cc:2454] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:51:56.661545 18870 raft_consensus.cc:686] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:56.664047 19105 raft_consensus.cc:686] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 5 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:51:56.698733 16450 leader_election.cc:302] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:56.578169 18827 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:56.684604 19252 raft_consensus.cc:2781] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:51:56.676690 19198 raft_consensus.cc:482] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:56.683107 19190 raft_consensus.cc:3037] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:56.707921 16450 leader_election.cc:302] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:56.711416 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:56.721683 16450 leader_election.cc:302] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:56.732952 16743 raft_consensus.cc:2379] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 4: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:51:56.734097 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:56.735877 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:56.714120 19244 raft_consensus.cc:2781] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:56.677414 17727 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:56.722352 18730 raft_consensus.cc:2781] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:56.747932 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:56.748636 16624 raft_consensus.cc:1799] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Handling vote request from an unknown peer d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:56.753070 16624 raft_consensus.cc:2396] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 3 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260810 18:51:56.751004 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:56.753611 16749 raft_consensus.cc:2454] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:56.754927 18870 consensus_queue.cc:237] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:56.757041 19099 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:56.751931 16695 leader_election.cc:302] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:56.723982 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:56.724789 19252 raft_consensus.cc:686] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 4 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:51:56.687505 18852 consensus_queue.cc:237] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:56.776139 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:56.776829 16506 raft_consensus.cc:2454] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
I20260810 18:51:56.777042 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:56.778187 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:56.779774 16748 raft_consensus.cc:1261] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:51:56.780730 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:56.781780 19291 raft_consensus.cc:2781] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:51:56.782300 19291 raft_consensus.cc:686] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:51:56.783315 19291 consensus_queue.cc:237] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:56.759577 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:56.752615 19114 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:56.787303 16621 raft_consensus.cc:2454] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:56.787588 18903 raft_consensus.cc:2781] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:51:56.788069 18903 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:56.788394 18903 raft_consensus.cc:3037] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:56.789088 19114 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:56.789692 17770 raft_consensus.cc:2454] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:56.789798 16448 leader_election.cc:302] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:56.792524 18728 raft_consensus.cc:2781] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:51:56.792986 18728 raft_consensus.cc:482] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:56.793395 18728 raft_consensus.cc:3037] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:51:56.764645 16695 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:56.764755 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:56.745368 19244 raft_consensus.cc:482] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:56.767868 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:56.768244 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:56.768172 19105 consensus_queue.cc:237] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:56.769004 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:56.769556 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:56.834390 16628 raft_consensus.cc:3037] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:56.770594 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:56.835247 16504 raft_consensus.cc:2379] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 3: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:56.770980 16632 raft_consensus.cc:1261] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260810 18:51:56.771083 16747 raft_consensus.cc:1261] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:51:56.771766 16501 raft_consensus.cc:1261] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:51:56.772586 19099 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:56.773094 19073 consensus_queue.cc:1035] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:51:56.799080 16750 raft_consensus.cc:2354] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for earlier term 2. Current term is 3.
I20260810 18:51:56.767513 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:51:56.753531 19172 consensus_meta.cc:287] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.632s user 0.010s sys 0.008s
I20260810 18:51:56.765662 18730 raft_consensus.cc:686] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 2 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:56.808068 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:56.859699 16749 raft_consensus.cc:3037] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
W20260810 18:51:56.813162 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 48596.6us
I20260810 18:51:56.861508 16507 raft_consensus.cc:1226] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3. Ops: []
I20260810 18:51:56.861618 16695 leader_election.cc:302] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:56.698367 18809 consensus_queue.cc:260] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:56.813987 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:56.866530 16743 raft_consensus.cc:2454] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:56.721609 19156 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:56.814973 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:51:56.820677 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 162078us
I20260810 18:51:56.871398 16752 raft_consensus.cc:1799] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Handling vote request from an unknown peer d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:56.871770 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:56.824334 16448 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:56.872305 16752 raft_consensus.cc:2396] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 3 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
W20260810 18:51:56.872663 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 48450.3us
I20260810 18:51:56.827883 16450 leader_election.cc:302] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:56.842588 16624 raft_consensus.cc:1261] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260810 18:51:56.814388 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 152850us
I20260810 18:51:56.803213 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:56.853206 19307 raft_consensus.cc:482] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:56.855265 18922 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:56.855352 19312 raft_consensus.cc:482] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:56.814913 19244 raft_consensus.cc:3037] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:56.862716 16695 leader_election.cc:302] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:56.888761 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:56.888942 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:56.835132 16500 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
W20260810 18:51:56.868570 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 275526us
I20260810 18:51:56.869443 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:56.833228 19135 raft_consensus.cc:2781] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:51:56.721974 19198 raft_consensus.cc:504] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:56.900928 19198 leader_election.cc:288] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:56.857970 16742 raft_consensus.cc:2379] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 5: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:56.877152 19065 raft_consensus.cc:2781] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:56.901674 19065 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:56.902077 19065 raft_consensus.cc:3037] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:56.902153 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:56.902894 16742 raft_consensus.cc:2454] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:56.903916 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:56.905205 16495 raft_consensus.cc:2379] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 4: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:56.905102 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:56.905885 16742 raft_consensus.cc:2379] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 4: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:51:56.906224 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:56.836280 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:56.906693 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:56.907351 16742 raft_consensus.cc:3037] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:56.907621 18808 raft_consensus.cc:2726] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:51:56.894932 16447 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:56.906845 16752 raft_consensus.cc:2362] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2. Re-sending same reply.
I20260810 18:51:56.872514 16747 raft_consensus.cc:2454] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
W20260810 18:51:56.876389 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 48650.9us
I20260810 18:51:56.878082 16748 raft_consensus.cc:2454] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:56.881639 16749 raft_consensus.cc:2454] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:51:56.882141 17726 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:51:56.883275 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:51:56.884277 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 330547us
I20260810 18:51:56.913518 16573 leader_election.cc:302] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:51:56.878890 16636 raft_consensus.cc:2454] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:51:56.846731 16740 raft_consensus.cc:2454] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:51:56.889763 16743 raft_consensus.cc:2454] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:56.916800 16631 raft_consensus.cc:2454] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:51:56.865820 19099 raft_consensus.cc:2781] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:56.858425 18931 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:56.814576 19248 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:56.893976 16504 raft_consensus.cc:3037] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:51:56.931602 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:56.947873 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:56.948191 16749 raft_consensus.cc:2454] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:56.894850 18827 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:56.895685 16624 raft_consensus.cc:1261] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260810 18:51:56.896687 18957 raft_consensus.cc:504] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:56.897025 18922 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:56.897652 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:56.894906 18809 raft_consensus.cc:3037] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:56.875003 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:56.967432 16750 raft_consensus.cc:2362] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4. Re-sending same reply.
I20260810 18:51:56.918228 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:56.918861 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:56.929483 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:56.930648 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:51:56.932137 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 50107.9us
I20260810 18:51:56.949756 19099 raft_consensus.cc:482] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:56.893340 19307 raft_consensus.cc:504] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:56.951713 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:56.978183 16623 raft_consensus.cc:2379] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 2: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:56.932857 16573 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:56.955865 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:56.979390 16752 raft_consensus.cc:2379] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 4: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:51:56.980742 18730 consensus_queue.cc:237] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:56.957067 19327 raft_consensus.cc:2781] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:51:56.981971 19327 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:56.982288 19327 raft_consensus.cc:3037] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:56.984333 18957 leader_election.cc:288] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:56.978262 19099 raft_consensus.cc:3037] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:56.931010 16746 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:51:56.861150 19172 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:56.960316 16743 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b"
dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
mode: GRACEFUL
new_leader_uuid: "d304ac47fdf846de81337057a4ef7bea"
from {username='slave'} at 127.0.0.1:45870
I20260810 18:51:56.987088 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:56.988287 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:56.988415 19172 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:56.973871 19018 raft_consensus.cc:2726] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791) with higher term. Message: Invalid argument: T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for earlier term 2. Current term is 3.
I20260810 18:51:56.898490 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:56.959262 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:56.989526 16623 raft_consensus.cc:1226] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 2. Ops: []
I20260810 18:51:56.959831 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:56.990480 16747 raft_consensus.cc:2454] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:56.960170 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:56.991261 16748 raft_consensus.cc:1799] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Handling vote request from an unknown peer 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:51:56.991757 16748 raft_consensus.cc:2379] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 3: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:51:56.989552 17785 raft_consensus.cc:2379] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 4: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:56.975503 16631 raft_consensus.cc:2379] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 5: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:56.980764 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:56.964744 19279 raft_consensus.cc:2726] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:51:56.996341 16502 raft_consensus.cc:2379] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 3: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:56.997336 16502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56468 (request call id 100) took 1097 ms. Trace:
I20260810 18:51:56.997685 16502 rpcz_store.cc:276] 0810 18:51:55.900127 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:55.943210 (+ 43083us) service_pool.cc:227] Handling call
0810 18:51:56.997146 (+1053936us) spinlock_profiling.cc:243] Waited 863 ms on lock 0x7b54001ff4d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:56.997281 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1985181568}
I20260810 18:51:56.932197 19198 raft_consensus.cc:2781] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:51:56.999312 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:57.000025 16624 raft_consensus.cc:2379] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 4: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:56.887349 18948 consensus_queue.cc:1035] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.070s
I20260810 18:51:57.000762 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:56.959731 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:57.001462 16624 raft_consensus.cc:2379] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 3: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:57.001576 16749 raft_consensus.cc:2454] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:51:57.002197 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:57.002569 17786 raft_consensus.cc:2454] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:51:56.965147 19250 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:56.989764 16621 raft_consensus.cc:2379] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 2: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:51:56.977044 17771 raft_consensus.cc:3037] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:57.003585 16633 raft_consensus.cc:2454] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:51:57.004472 18922 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:57.004503 17781 raft_consensus.cc:1226] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3. Ops: []
I20260810 18:51:56.977101 16628 raft_consensus.cc:2454] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:51:57.009099 19248 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:57.012743 19307 leader_election.cc:288] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:56.893100 19312 raft_consensus.cc:504] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:57.014986 19328 log.cc:924] Time spent T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.159s user 0.000s sys 0.001s
I20260810 18:51:57.054510 16504 raft_consensus.cc:2454] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:51:57.015156 18808 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:51:57.015967 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 83219.3us
I20260810 18:51:57.017064 16701 leader_election.cc:302] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:57.058169 16573 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:51:57.059502 16573 leader_election.cc:398] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Vote denied by peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387) with higher term. Message: Invalid argument: T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 3. Current term is 4.
I20260810 18:51:57.059850 16573 leader_election.cc:401] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260810 18:51:57.020172 16632 raft_consensus.cc:2454] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 2.
I20260810 18:51:57.022064 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:51:57.002743 16624 raft_consensus.cc:2454] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
I20260810 18:51:56.909972 18852 consensus_queue.cc:1035] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.065s
W20260810 18:51:57.061313 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 166486us
I20260810 18:51:57.063563 17724 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:57.065639 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:57.053539 16448 leader_election.cc:302] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:57.018282 18957 raft_consensus.cc:3037] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:57.079495 16448 leader_election.cc:302] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:57.072690 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:57.074297 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:51:56.972770 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 78258.9us
I20260810 18:51:57.035307 18827 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:57.050318 19129 raft_consensus.cc:2781] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:57.088728 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:57.034436 19198 raft_consensus.cc:686] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
W20260810 18:51:57.092176 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 174869us
I20260810 18:51:57.102433 16624 raft_consensus.cc:1226] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3. Ops: []
I20260810 18:51:57.103615 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:57.061050 19248 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:57.072166 19307 raft_consensus.cc:2781] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260810 18:51:57.116685 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 99708.8us
I20260810 18:51:57.117645 16701 leader_election.cc:302] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:57.111348 19248 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:57.112340 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:56.956617 19252 consensus_queue.cc:237] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:57.050159 19018 consensus_queue.cc:1035] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20260810 18:51:57.091796 18922 raft_consensus.cc:2726] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387) with higher term. Message: Invalid argument: T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 3. Current term is 4.
I20260810 18:51:57.091459 19073 consensus_queue.cc:1035] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260810 18:51:57.121029 16701 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:57.099957 19162 raft_consensus.cc:2781] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:51:57.098785 19291 raft_consensus.cc:3037] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:57.099236 18931 raft_consensus.cc:2726] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:51:57.077368 18808 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:57.123342 19307 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:57.081274 19065 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:57.165203 16495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56468 (request call id 112) took 1032 ms. Trace:
I20260810 18:51:57.165632 16575 leader_election.cc:398] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Vote denied by peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171) with higher term. Message: Invalid argument: T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 2. Current term is 3.
I20260810 18:51:57.168128 19198 consensus_queue.cc:237] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:57.132427 19130 raft_consensus.cc:2726] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:51:57.171810 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:57.091169 19163 consensus_queue.cc:1046] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.067s
I20260810 18:51:57.156219 18631 raft_consensus.cc:482] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:57.160259 19162 raft_consensus.cc:686] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:57.178763 18631 raft_consensus.cc:504] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:57.179571 19162 consensus_queue.cc:237] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:57.181053 18631 leader_election.cc:288] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:57.123802 19099 raft_consensus.cc:504] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:57.184327 16504 raft_consensus.cc:2354] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for earlier term 2. Current term is 3.
I20260810 18:51:57.187724 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:57.191843 16507 raft_consensus.cc:2379] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 5: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:51:57.170692 19307 raft_consensus.cc:3037] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:57.194877 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:57.157521 19304 raft_consensus.cc:2781] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:51:57.162611 19156 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:57.172153 18931 raft_consensus.cc:2726] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:51:57.254081 16743 raft_consensus.cc:595] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Received request to transfer leadership to TS d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:57.254609 16743 raft_consensus.cc:603] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260810 18:51:57.108425 18827 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:51:57.249972 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:57.250128 16447 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:57.184952 18948 raft_consensus.cc:2781] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:51:57.260643 16573 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
W20260810 18:51:57.264448 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 143143us
I20260810 18:51:57.172161 19279 consensus_queue.cc:1035] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.082s
I20260810 18:51:57.274463 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:57.275290 16746 raft_consensus.cc:2362] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2. Re-sending same reply.
I20260810 18:51:57.276446 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:57.277068 16746 raft_consensus.cc:2454] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:51:57.283139 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:57.172950 16495 rpcz_store.cc:276] 0810 18:51:56.133040 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:56.136913 (+ 3873us) service_pool.cc:227] Handling call
0810 18:51:57.164989 (+1028076us) spinlock_profiling.cc:243] Waited 634 ms on lock 0x7b54001e14d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:57.165135 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1457945856}
I20260810 18:51:57.270610 19339 raft_consensus.cc:2781] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:51:57.292650 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:57.293753 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:57.295634 16752 raft_consensus.cc:3037] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:57.297034 16750 raft_consensus.cc:2454] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
W20260810 18:51:57.300607 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 200696us
W20260810 18:51:57.305228 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 44683.4us
W20260810 18:51:57.308719 19333 log.cc:924] Time spent T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.189s user 0.001s sys 0.000s
I20260810 18:51:57.288726 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:57.366377 16573 leader_election.cc:302] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:57.366760 16450 leader_election.cc:398] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Vote denied by peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171) with higher term. Message: Invalid argument: T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for earlier term 2. Current term is 3.
I20260810 18:51:57.367662 16752 raft_consensus.cc:2454] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
W20260810 18:51:57.368614 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 118709us
I20260810 18:51:57.400007 18948 raft_consensus.cc:686] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:57.398866 16450 leader_election.cc:401] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260810 18:51:57.400530 16575 leader_election.cc:302] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:57.285197 18922 consensus_queue.cc:1046] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
W20260810 18:51:57.402282 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 322850us
W20260810 18:51:57.406437 18728 consensus_meta.cc:287] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.613s user 0.012s sys 0.000s
I20260810 18:51:57.306679 19364 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:57.313119 19351 raft_consensus.cc:2726] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:51:57.369867 19250 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:57.384052 19339 raft_consensus.cc:686] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:57.423620 16632 raft_consensus.cc:1261] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260810 18:51:57.407744 19312 leader_election.cc:288] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:57.306424 19327 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:57.476907 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:57.414541 18827 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:51:57.421587 19289 consensus_meta.cc:287] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.941s user 0.014s sys 0.000s
I20260810 18:51:57.422072 16742 raft_consensus.cc:2396] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260810 18:51:57.480866 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 80656.1us
I20260810 18:51:57.416872 18631 raft_consensus.cc:482] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:57.417007 19065 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:57.423502 19248 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:57.487543 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:51:57.503443 16632 raft_consensus.cc:3037] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
W20260810 18:51:57.505831 19352 log.cc:924] Time spent T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.095s user 0.000s sys 0.001s
I20260810 18:51:57.505764 17771 raft_consensus.cc:2454] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:51:57.499589 19135 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:57.506819 19135 raft_consensus.cc:3037] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:51:57.497632 16575 leader_election.cc:302] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:57.501164 19338 raft_consensus.cc:3037] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:57.482846 19351 raft_consensus.cc:2781] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260810 18:51:57.502486 19343 log.cc:924] Time spent T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.260s user 0.002s sys 0.002s
I20260810 18:51:57.414431 18948 consensus_queue.cc:237] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:57.502871 19297 log.cc:924] Time spent T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
W20260810 18:51:57.507161 19341 log.cc:924] Time spent T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.236s user 0.002s sys 0.002s
I20260810 18:51:57.507246 19354 raft_consensus.cc:482] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:51:57.510742 19244 consensus_meta.cc:287] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.609s user 0.008s sys 0.000s
I20260810 18:51:57.487547 19289 raft_consensus.cc:504] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:57.513705 18957 raft_consensus.cc:2726] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 18:51:57.514830 18975 consensus_meta.cc:287] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 1.026s user 0.014s sys 0.011s
I20260810 18:51:57.423779 19163 raft_consensus.cc:482] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:51:57.515944 18903 consensus_meta.cc:287] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.727s user 0.011s sys 0.000s
I20260810 18:51:57.516294 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:57.516623 19327 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:57.517477 17786 raft_consensus.cc:2379] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 4: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:51:57.517861 16632 raft_consensus.cc:2454] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:51:57.517447 16695 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
W20260810 18:51:57.586609 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 69273.6us
I20260810 18:51:57.529574 16575 leader_election.cc:302] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
W20260810 18:51:57.525875 19197 consensus_meta.cc:287] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.937s user 0.017s sys 0.000s
I20260810 18:51:57.551653 19351 raft_consensus.cc:482] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:57.530970 19353 consensus_queue.cc:1035] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:51:57.416394 18728 raft_consensus.cc:504] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:57.501243 19339 consensus_queue.cc:237] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:57.484782 19312 raft_consensus.cc:2781] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:57.554401 19129 raft_consensus.cc:482] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:57.559355 19130 raft_consensus.cc:3032] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:51:57.563704 19307 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:57.489018 19099 leader_election.cc:288] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:57.599552 19130 raft_consensus.cc:729] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:57.605561 19130 consensus_queue.cc:260] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:57.525852 18631 raft_consensus.cc:504] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:57.606635 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:57.607630 16624 raft_consensus.cc:2379] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 4: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:57.585219 16623 raft_consensus.cc:3037] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:57.607966 18631 leader_election.cc:288] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:57.579968 18903 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:57.424042 19364 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:57.609428 19312 raft_consensus.cc:686] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 2 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:57.609915 18903 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:57.610560 16745 raft_consensus.cc:1226] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 3. Current term is 4. Ops: []
I20260810 18:51:57.610415 19312 consensus_queue.cc:237] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:57.611503 19364 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:57.572173 16628 raft_consensus.cc:1261] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260810 18:51:57.599504 16500 consensus_meta.cc:287] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.691s user 0.009s sys 0.001s
I20260810 18:51:57.601096 16747 raft_consensus.cc:1226] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 2. Current term is 3. Ops: []
I20260810 18:51:57.613169 16500 raft_consensus.cc:2454] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:51:57.613873 16508 raft_consensus.cc:1226] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 2. Current term is 3. Ops: []
I20260810 18:51:57.606086 19307 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:51:57.615428 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 311867us
I20260810 18:51:57.618208 16450 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:57.591321 18931 raft_consensus.cc:2781] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:51:57.628448 16344 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37030 (request call id 202) took 7148 ms. Trace:
W20260810 18:51:57.633991 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 221673us
I20260810 18:51:57.635478 16747 raft_consensus.cc:1261] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260810 18:51:57.594960 19351 raft_consensus.cc:3037] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:57.640921 16573 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:57.597744 18827 raft_consensus.cc:482] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:57.599931 19129 raft_consensus.cc:3037] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:57.578990 19244 raft_consensus.cc:504] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:57.599694 18809 raft_consensus.cc:482] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:57.593170 19163 raft_consensus.cc:504] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:51:57.614814 19130 raft_consensus.cc:3037] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:57.595549 18957 consensus_queue.cc:1035] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
I20260810 18:51:57.669498 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:57.643121 18631 consensus_queue.cc:1035] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260810 18:51:57.670125 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:57.671329 16506 raft_consensus.cc:2396] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260810 18:51:57.671870 18957 raft_consensus.cc:2781] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:51:57.649718 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:57.673236 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:57.673703 19289 leader_election.cc:288] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:57.673791 16506 raft_consensus.cc:3037] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:57.678929 16748 raft_consensus.cc:3037] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:57.588654 19197 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:57.683619 19197 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:57.684000 16492 tablet_service.cc:1514] Processing DeleteTablet for tablet 03cb267b222b440981c94d431e7273bf with delete_type TABLET_DATA_TOMBSTONED (TS d304ac47fdf846de81337057a4ef7bea not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52330
I20260810 18:51:57.684532 16622 raft_consensus.cc:2379] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 3: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:57.649591 18809 raft_consensus.cc:504] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:57.631369 18903 raft_consensus.cc:482] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:57.651276 18728 leader_election.cc:288] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:57.628731 19367 raft_consensus.cc:2726] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:51:57.588881 18975 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:57.646958 16695 leader_election.cc:302] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:57.652851 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:57.656967 16742 raft_consensus.cc:1261] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260810 18:51:57.613715 19365 consensus_queue.cc:1046] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.056s
I20260810 18:51:57.649377 18827 raft_consensus.cc:504] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:57.659514 16623 raft_consensus.cc:2454] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:51:57.631685 19307 raft_consensus.cc:2726] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:51:57.643522 18931 raft_consensus.cc:686] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:57.664664 17727 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:57.725299 18942 raft_consensus.cc:3032] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:51:57.725700 18942 raft_consensus.cc:729] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
W20260810 18:51:57.725939 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 61390.1us
I20260810 18:51:57.726558 18942 consensus_queue.cc:260] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:57.713649 16500 raft_consensus.cc:3037] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 4
I20260810 18:51:57.664947 19350 raft_consensus.cc:482] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:57.665872 19291 raft_consensus.cc:2726] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:51:57.728442 19291 raft_consensus.cc:2726] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260810 18:51:57.728259 19350 raft_consensus.cc:504] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:57.669351 19190 consensus_meta.cc:287] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.927s user 0.012s sys 0.000s
I20260810 18:51:57.665966 19172 raft_consensus.cc:482] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:57.729507 19190 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:57.730494 18809 leader_election.cc:288] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:57.730718 16626 raft_consensus.cc:2379] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 4: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:57.667665 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:57.704120 16501 raft_consensus.cc:2354] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 2. Current term is 3.
I20260810 18:51:57.731210 19304 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:57.731683 19304 raft_consensus.cc:3037] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:57.732690 16626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 119) took 1171 ms. Trace:
I20260810 18:51:57.733047 16626 rpcz_store.cc:276] 0810 18:51:56.561495 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:56.563015 (+ 1520us) service_pool.cc:227] Handling call
0810 18:51:57.732507 (+1169492us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54002a5654. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:57.732634 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2645553536}
I20260810 18:51:57.733896 19380 consensus_queue.cc:1035] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260810 18:51:57.733919 18808 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:57.582027 19354 raft_consensus.cc:504] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:57.705642 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:57.735769 19244 leader_election.cc:288] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:57.693454 18957 raft_consensus.cc:686] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:57.729058 19291 consensus_queue.cc:1046] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.008s
W20260810 18:51:57.705992 19379 log.cc:924] Time spent T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.112s user 0.000s sys 0.000s
I20260810 18:51:57.737329 19291 raft_consensus.cc:3032] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:51:57.736939 18957 consensus_queue.cc:237] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:57.737668 19291 raft_consensus.cc:729] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:51:57.738091 19163 leader_election.cc:288] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:57.738472 19291 consensus_queue.cc:260] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:57.739744 19291 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:57.739961 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:57.740626 16742 raft_consensus.cc:3037] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
W20260810 18:51:57.713228 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 66364.8us
I20260810 18:51:57.743220 16695 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:57.744010 16695 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:57.715320 16631 raft_consensus.cc:1226] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 1. Current term is 3. Ops: []
I20260810 18:51:57.714586 16747 raft_consensus.cc:1261] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260810 18:51:57.725159 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:57.712437 18903 raft_consensus.cc:504] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:57.752594 18903 leader_election.cc:288] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:57.753168 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:57.753964 16750 raft_consensus.cc:2379] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 4: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:51:57.696094 16748 raft_consensus.cc:1261] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260810 18:51:57.730190 19172 raft_consensus.cc:504] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:57.736631 19354 leader_election.cc:288] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:57.705260 19392 raft_consensus.cc:2726] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:51:57.772274 16750 raft_consensus.cc:1261] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260810 18:51:57.711128 19338 raft_consensus.cc:2726] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171) with higher term. Message: Invalid argument: T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for earlier term 2. Current term is 3.
I20260810 18:51:57.712604 18975 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:51:57.815289 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 285808us
I20260810 18:51:57.727774 18942 raft_consensus.cc:3037] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:57.767679 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:57.738766 16342 catalog_manager.cc:5495] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 1 to 2, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196). New cstate: current_term: 2 leader_uuid: "831925dd8b644ebab7eb0d672a94eddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:51:57.747577 17785 raft_consensus.cc:2454] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
W20260810 18:51:57.693977 19162 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.010s sys 0.031s
I20260810 18:51:57.826682 16743 raft_consensus.cc:3037] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:57.693373 16506 raft_consensus.cc:2454] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
W20260810 18:51:57.774279 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 379316us
I20260810 18:51:57.749074 19135 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:57.767143 18808 raft_consensus.cc:2781] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:51:57.829660 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:57.830525 16750 raft_consensus.cc:3037] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:57.833971 16507 raft_consensus.cc:1261] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260810 18:51:57.835423 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:57.836205 16507 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 4.
I20260810 18:51:57.768018 19163 raft_consensus.cc:2781] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:51:57.633848 16344 rpcz_store.cc:276] 0810 18:51:50.479601 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:50.479726 (+ 125us) service_pool.cc:227] Handling call
0810 18:51:57.628353 (+7148627us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 18:51:55.832265 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 18:51:55.857190 (+ 24925us) write_op.cc:432] Acquiring schema lock in shared mode
0810 18:51:55.857254 (+ 64us) write_op.cc:435] Acquired schema lock
0810 18:51:55.857292 (+ 38us) tablet.cc:660] Decoding operations
0810 18:51:55.858075 (+ 783us) write_op.cc:620] Acquiring the partition lock for write op
0810 18:51:55.858175 (+ 100us) write_op.cc:641] Partition lock acquired for write op
0810 18:51:55.858220 (+ 45us) tablet.cc:683] Acquiring locks for 3 operations
0810 18:51:55.858710 (+ 490us) tablet.cc:699] Row locks acquired
0810 18:51:55.858749 (+ 39us) write_op.cc:260] PREPARE: finished
0810 18:51:55.858950 (+ 201us) write_op.cc:270] Start()
0810 18:51:55.859153 (+ 203us) write_op.cc:276] Timestamp: P: 1786387915858858 usec, L: 0
0810 18:51:55.859190 (+ 37us) op_driver.cc:331] REPLICATION: starting
0810 18:51:55.859877 (+ 687us) log.cc:842] Serialized 1307 byte log entry
0810 18:51:56.554397 (+694520us) op_driver.cc:444] REPLICATION: finished
0810 18:51:56.855993 (+301596us) write_op.cc:301] APPLY: starting
0810 18:51:56.856156 (+ 163us) tablet.cc:1365] starting BulkCheckPresence
0810 18:51:56.856399 (+ 243us) tablet.cc:1368] finished BulkCheckPresence
0810 18:51:56.864787 (+ 8388us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 18:51:56.865480 (+ 693us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 18:51:56.865783 (+ 303us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0810 18:51:56.865860 (+ 77us) write_op.cc:312] APPLY: finished
0810 18:51:56.895439 (+ 29579us) log.cc:842] Serialized 35 byte log entry
0810 18:51:56.895636 (+ 197us) write_op.cc:489] Releasing partition, row and schema locks
0810 18:51:56.895748 (+ 112us) write_op.cc:454] Released schema lock
0810 18:51:56.895808 (+ 60us) write_op.cc:341] FINISH: Updating metrics
0810 18:51:56.896085 (+ 277us) write_op.cc:489] Releasing partition, row and schema locks
0810 18:51:56.896112 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":11,"child_traces":[["op",{"apply.queue_time_us":259475,"apply.run_cpu_time_us":3866,"apply.run_wall_time_us":52728,"num_ops":3,"prepare.queue_time_us":328986,"prepare.run_cpu_time_us":77892,"prepare.run_wall_time_us":366429,"raft.queue_time_us":250194,"raft.run_cpu_time_us":55523,"raft.run_wall_time_us":113857,"replication_time_us":695118,"thread_start_us":616822,"threads_started":4,"wal-append.queue_time_us":425147}]]}
I20260810 18:51:57.823266 19073 consensus_queue.cc:1035] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20260810 18:51:57.837131 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:57.841257 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:57.816964 18903 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:51:57.856698 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 214060us
I20260810 18:51:57.847358 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:51:57.838426 19162 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.656s user 0.011s sys 0.032s
I20260810 18:51:57.840590 16575 leader_election.cc:302] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:57.843386 16695 leader_election.cc:302] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:57.859516 17724 leader_election.cc:398] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Vote denied by peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791) with higher term. Message: Invalid argument: T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for earlier term 2. Current term is 3.
I20260810 18:51:57.865586 17724 leader_election.cc:401] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260810 18:51:57.858153 16507 raft_consensus.cc:2379] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 4: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:57.863801 16508 raft_consensus.cc:2454] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:51:57.841142 16342 catalog_manager.cc:5495] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: config changed from index 3 to 4, term changed from 1 to 2, leader changed from d304ac47fdf846de81337057a4ef7bea (127.14.32.193) to <none>, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New cstate: current_term: 2 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:57.863471 16506 raft_consensus.cc:3037] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:57.871275 16504 raft_consensus.cc:2454] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:51:57.872622 16750 raft_consensus.cc:2454] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:51:57.866626 17724 leader_election.cc:302] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:57.842021 18808 raft_consensus.cc:686] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:57.885399 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:57.899082 16750 raft_consensus.cc:2379] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 2: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:57.842270 18827 leader_election.cc:288] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:57.838703 19129 raft_consensus.cc:504] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:57.844079 19163 raft_consensus.cc:686] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:57.889178 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:57.889653 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:57.889997 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:57.890576 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:57.873019 18931 consensus_queue.cc:237] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:57.889104 17770 raft_consensus.cc:1226] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 4 LEADER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 3. Current term is 4. Ops: []
I20260810 18:51:57.884388 19099 raft_consensus.cc:2781] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:51:57.895876 16575 leader_election.cc:302] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:57.919710 16575 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:57.920814 16575 leader_election.cc:302] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:57.896533 19312 raft_consensus.cc:2726] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:51:57.842203 19350 leader_election.cc:288] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:57.842895 19382 raft_consensus.cc:482] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:57.846007 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:57.874840 16342 catalog_manager.cc:5495] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 1 to 2, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 2 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:57.838635 19244 consensus_queue.cc:1035] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20260810 18:51:57.847962 19114 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:51:57.879134 19380 raft_consensus.cc:3037] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:57.902076 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:57.902215 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:57.929531 16342 catalog_manager.cc:5495] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 1 to 2, leader changed from d304ac47fdf846de81337057a4ef7bea (127.14.32.193) to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 2 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:51:57.850670 19105 consensus_queue.cc:1035] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20260810 18:51:57.885532 19190 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:57.839375 19367 consensus_queue.cc:1035] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
I20260810 18:51:57.866946 18903 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:57.848513 19172 leader_election.cc:288] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:57.922848 16573 leader_election.cc:398] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Vote denied by peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791) with higher term. Message: Invalid argument: T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 2. Current term is 3.
I20260810 18:51:57.948658 16573 leader_election.cc:401] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260810 18:51:57.925951 16752 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d"
dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
mode: GRACEFUL
new_leader_uuid: "d304ac47fdf846de81337057a4ef7bea"
from {username='slave'} at 127.0.0.1:45870
I20260810 18:51:57.926003 16450 leader_election.cc:302] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:57.926999 16745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43726 (request call id 1315) took 1169 ms. Trace:
I20260810 18:51:57.949577 16745 rpcz_store.cc:276] 0810 18:51:56.752604 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:56.761848 (+ 9244us) service_pool.cc:227] Handling call
0810 18:51:56.767816 (+ 5968us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:57.610915 (+843099us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:57.921444 (+310529us) spinlock_profiling.cc:243] Waited 842 ms on lock 0x7b540020ab54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:51:57.921647 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1937676544}
I20260810 18:51:57.908282 17781 raft_consensus.cc:1261] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260810 18:51:57.930478 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:57.900142 16743 raft_consensus.cc:2454] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:51:57.882229 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:57.952529 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:57.953198 16507 raft_consensus.cc:3037] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:57.954483 16695 leader_election.cc:302] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:57.934110 16750 raft_consensus.cc:2454] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:51:57.947158 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:57.945184 16342 catalog_manager.cc:5495] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 1 to 2, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196). New cstate: current_term: 2 leader_uuid: "831925dd8b644ebab7eb0d672a94eddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:51:57.941890 18631 raft_consensus.cc:2726] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:51:57.924520 16740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1402) took 1104 ms. Trace:
I20260810 18:51:57.957919 16450 leader_election.cc:302] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:57.958261 18808 consensus_queue.cc:237] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:57.958016 16740 rpcz_store.cc:276] 0810 18:51:56.819772 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:57.016630 (+196858us) service_pool.cc:227] Handling call
0810 18:51:57.076681 (+ 60051us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:57.102131 (+ 25450us) raft_consensus.cc:1520] Early marking committed up to index 3
0810 18:51:57.307446 (+205315us) log.cc:842] Serialized 14 byte log entry
0810 18:51:57.900439 (+592993us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:57.904317 (+ 3878us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:51:57.910395 (+ 6078us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:57.914970 (+ 4575us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:57.920269 (+ 5299us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32850,"prepare.queue_time_us":65690,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":39313,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":50111,"thread_start_us":605581,"threads_started":2}
I20260810 18:51:57.959445 16743 raft_consensus.cc:1261] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260810 18:51:57.961884 16500 raft_consensus.cc:2454] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 4.
I20260810 18:51:57.962574 16575 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:57.924382 19198 raft_consensus.cc:2726] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:51:57.956501 16508 raft_consensus.cc:2454] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:51:57.964779 17724 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:51:57.971802 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 49143.1us
I20260810 18:51:57.974089 16506 raft_consensus.cc:1261] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260810 18:51:57.976015 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:57.947788 17775 raft_consensus.cc:2379] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 3: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
W20260810 18:51:57.981490 16622 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 1381) took 1246 ms (client timeout 1558 ms). Trace:
I20260810 18:51:57.980674 16573 leader_election.cc:302] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:57.982446 16505 raft_consensus.cc:2379] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 4: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:51:57.985684 16450 leader_election.cc:302] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:57.965765 17780 raft_consensus.cc:2379] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 4: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:51:57.988530 16575 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:57.986593 16507 raft_consensus.cc:2454] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
W20260810 18:51:57.986033 16622 rpcz_store.cc:269] 0810 18:51:56.735374 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:56.735552 (+ 178us) service_pool.cc:227] Handling call
0810 18:51:57.981229 (+1245677us) spinlock_profiling.cc:243] Waited 948 ms on lock 0x7b54003859d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:57.981419 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2179321600}
I20260810 18:51:57.970903 19077 raft_consensus.cc:2781] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:51:57.970697 18922 raft_consensus.cc:2781] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:57.977835 19198 raft_consensus.cc:3037] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 3
I20260810 18:51:58.009948 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:58.027942 16508 raft_consensus.cc:2379] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 4: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:51:57.908502 19390 consensus_queue.cc:1035] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.076s
I20260810 18:51:58.023218 18957 raft_consensus.cc:2781] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:51:57.946941 19114 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:57.996239 19339 raft_consensus.cc:2781] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:58.007975 19385 raft_consensus.cc:2781] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:51:58.021485 19291 raft_consensus.cc:2781] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:51:58.026463 19312 raft_consensus.cc:2781] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:58.042672 16573 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:57.970459 19351 raft_consensus.cc:504] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:58.025643 19386 raft_consensus.cc:2726] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:51:58.023626 18922 raft_consensus.cc:482] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:58.069762 16573 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:57.932125 19382 raft_consensus.cc:504] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:58.032622 18827 raft_consensus.cc:2781] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:51:58.034562 18957 raft_consensus.cc:482] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:58.032758 19077 raft_consensus.cc:686] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:58.044039 19339 raft_consensus.cc:482] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:58.042073 19248 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
W20260810 18:51:58.121160 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 132734us
I20260810 18:51:58.026262 19327 consensus_queue.cc:1035] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20260810 18:51:58.129940 16752 raft_consensus.cc:595] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Received request to transfer leadership to TS d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:58.131898 16575 leader_election.cc:302] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:58.135226 16752 raft_consensus.cc:603] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260810 18:51:58.133273 19077 consensus_queue.cc:237] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:58.134897 16743 raft_consensus.cc:1226] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 2. Current term is 3. Ops: []
I20260810 18:51:58.070284 19099 raft_consensus.cc:686] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:51:58.078447 19291 raft_consensus.cc:686] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 4 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:51:58.118198 19307 raft_consensus.cc:2726] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:51:58.074185 19385 raft_consensus.cc:686] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:58.153786 17785 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55570 (request call id 364) took 1091 ms. Trace:
I20260810 18:51:58.120769 18957 raft_consensus.cc:3037] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:51:58.171332 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:58.173498 16743 raft_consensus.cc:3037] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:58.130143 16495 raft_consensus.cc:1261] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260810 18:51:58.134667 19392 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:58.130697 19339 raft_consensus.cc:3037] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:58.040571 19163 consensus_queue.cc:237] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:58.070669 19279 consensus_queue.cc:1035] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.072s
W20260810 18:51:58.130353 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 55107.8us
W20260810 18:51:58.185066 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 230667us
I20260810 18:51:58.174616 16631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1330) took 1181 ms. Trace:
I20260810 18:51:58.212216 16631 rpcz_store.cc:276] 0810 18:51:56.974209 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:56.994186 (+ 19977us) service_pool.cc:227] Handling call
0810 18:51:56.994475 (+ 289us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:57.748861 (+754386us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:58.154812 (+405951us) spinlock_profiling.cc:243] Waited 721 ms on lock 0x7b54002a4c54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:51:58.155031 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1657227648}
I20260810 18:51:58.129788 19397 raft_consensus.cc:2781] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260810 18:51:58.144718 19404 log.cc:924] Time spent T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.105s user 0.001s sys 0.000s
I20260810 18:51:58.168828 17785 rpcz_store.cc:276] 0810 18:51:57.062790 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:57.063164 (+ 374us) service_pool.cc:227] Handling call
0810 18:51:58.153558 (+1090394us) spinlock_profiling.cc:243] Waited 626 ms on lock 0x7b54005c03d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:58.153709 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1440314240}
I20260810 18:51:58.145505 19351 leader_election.cc:288] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:58.223121 16447 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:58.079134 18922 raft_consensus.cc:3037] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:58.182768 19340 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:58.134768 19312 raft_consensus.cc:686] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 2 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:58.147183 19129 leader_election.cc:288] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:58.175153 19365 raft_consensus.cc:2781] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:58.078828 18827 raft_consensus.cc:482] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:58.137415 19252 consensus_queue.cc:1035] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.074s
I20260810 18:51:58.231463 16743 raft_consensus.cc:2454] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:51:58.191485 19307 raft_consensus.cc:482] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:51:58.209622 19190 raft_consensus.cc:2781] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:51:58.174503 18870 consensus_queue.cc:1046] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20260810 18:51:58.139938 19248 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:51:58.233543 19402 log.cc:924] Time spent T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.269s user 0.000s sys 0.001s
I20260810 18:51:58.231098 19385 consensus_queue.cc:237] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:58.224743 19390 consensus_queue.cc:1035] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.010s
I20260810 18:51:58.222577 19392 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:58.230628 19289 raft_consensus.cc:2781] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:51:58.232681 16450 leader_election.cc:302] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:51:58.239194 19389 log.cc:924] Time spent T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.307s user 0.001s sys 0.000s
I20260810 18:51:58.231139 19365 raft_consensus.cc:482] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:58.240401 19198 raft_consensus.cc:2726] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791) with higher term. Message: Invalid argument: T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 2. Current term is 3.
I20260810 18:51:58.240770 19382 leader_election.cc:288] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:58.222721 18903 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:58.254088 17785 raft_consensus.cc:1261] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260810 18:51:58.264164 19190 raft_consensus.cc:686] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:58.220489 19397 raft_consensus.cc:482] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:51:58.224701 19403 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:51:58.232084 18827 raft_consensus.cc:3037] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
W20260810 18:51:58.260159 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 128361us
I20260810 18:51:58.279284 19365 raft_consensus.cc:3037] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
W20260810 18:51:58.269137 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 199448us
W20260810 18:51:58.265524 16742 consensus_meta.cc:287] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.524s user 0.009s sys 0.002s
W20260810 18:51:58.144936 19395 log.cc:924] Time spent T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.113s user 0.000s sys 0.001s
I20260810 18:51:58.268146 19380 raft_consensus.cc:2726] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791) with higher term. Message: Invalid argument: T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for earlier term 2. Current term is 3.
I20260810 18:51:58.305536 16575 leader_election.cc:302] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:58.230302 19340 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:58.291113 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:58.307096 19029 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:58.322269 19029 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:51:58.323149 19398 log.cc:924] Time spent T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.249s user 0.001s sys 0.000s
I20260810 18:51:58.324440 19248 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:58.324966 19029 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:51:58.327349 19401 log.cc:924] Time spent T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.275s user 0.000s sys 0.001s
I20260810 18:51:58.291640 19397 raft_consensus.cc:3037] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:58.328483 19312 consensus_queue.cc:237] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:58.230031 19291 consensus_queue.cc:237] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:51:58.264323 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 176244us
I20260810 18:51:58.278121 19289 raft_consensus.cc:686] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:58.282156 19162 raft_consensus.cc:2781] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:51:58.263538 19307 raft_consensus.cc:504] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:58.242067 19099 consensus_queue.cc:237] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:58.316797 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:58.336004 16573 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
W20260810 18:51:58.383410 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 77969.6us
I20260810 18:51:58.395383 16575 leader_election.cc:302] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:58.333402 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:58.290217 19393 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:51:58.312985 19304 consensus_meta.cc:287] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.581s user 0.010s sys 0.001s
I20260810 18:51:58.289690 19114 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:58.311692 19244 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:58.410983 16575 leader_election.cc:302] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
W20260810 18:51:58.412034 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 76138.2us
I20260810 18:51:58.318868 19135 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:58.292457 19129 consensus_queue.cc:1035] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.106s
I20260810 18:51:58.313256 16742 raft_consensus.cc:2454] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:51:58.413018 16573 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:58.387938 19432 raft_consensus.cc:2781] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260810 18:51:58.418906 19410 log.cc:924] Time spent T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.282s user 0.001s sys 0.000s
I20260810 18:51:58.414273 18728 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:58.411440 18975 raft_consensus.cc:2726] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260810 18:51:58.458501 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 47599.9us
I20260810 18:51:58.463339 18728 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:58.413663 19435 raft_consensus.cc:2781] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:58.485185 16575 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:58.450646 19198 raft_consensus.cc:2726] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:51:58.481940 19418 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:58.411911 19304 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:58.442026 19421 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:58.464960 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:58.422498 19244 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:58.455377 19432 raft_consensus.cc:482] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:58.501801 16575 leader_election.cc:302] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:58.487120 19435 raft_consensus.cc:482] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:51:58.505676 19435 raft_consensus.cc:3037] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:58.508489 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:58.510058 16495 raft_consensus.cc:3037] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:51:58.512140 18728 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:58.503446 19432 raft_consensus.cc:3037] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
W20260810 18:51:58.515436 19413 log.cc:924] Time spent T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.275s user 0.001s sys 0.000s
I20260810 18:51:58.517795 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:58.524021 16575 leader_election.cc:302] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:58.500569 19421 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:58.531997 19432 raft_consensus.cc:504] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:58.537075 19244 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:58.538477 19304 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:58.538661 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:58.482658 19289 consensus_queue.cc:237] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:58.539892 19406 tablet_replica.cc:330] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea: stopping tablet replica
I20260810 18:51:58.496940 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:51:58.519990 16500 raft_consensus.cc:1226] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 3. Current term is 4. Ops: []
I20260810 18:51:58.543557 19392 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:58.472273 19190 consensus_queue.cc:237] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:58.533012 18942 consensus_meta.cc:287] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.680s user 0.008s sys 0.002s
W20260810 18:51:58.519804 19423 log.cc:924] Time spent T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.234s user 0.001s sys 0.000s
I20260810 18:51:58.549856 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:58.555683 16742 raft_consensus.cc:2379] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 4: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:51:58.511026 19198 raft_consensus.cc:2781] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260810 18:51:58.581269 19425 log.cc:924] Time spent T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.249s user 0.001s sys 0.000s
I20260810 18:51:58.582595 19391 raft_consensus.cc:2781] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:51:58.583089 19432 leader_election.cc:288] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:58.583649 19391 raft_consensus.cc:686] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
W20260810 18:51:58.584115 19411 log.cc:924] Time spent T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.310s user 0.001s sys 0.000s
I20260810 18:51:58.407761 19393 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:58.602030 19338 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:58.602844 19338 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:58.609508 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:58.612108 19338 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:58.624120 19373 raft_consensus.cc:3032] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:51:58.624933 19373 raft_consensus.cc:729] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
W20260810 18:51:58.647784 16636 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1325) took 1732 ms (client timeout 1676 ms). Trace:
W20260810 18:51:58.648387 16636 rpcz_store.cc:269] 0810 18:51:56.915747 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:56.917912 (+ 2165us) service_pool.cc:227] Handling call
0810 18:51:58.647461 (+1729549us) spinlock_profiling.cc:243] Waited 901 ms on lock 0x7b540004d8d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:58.647679 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2071779584}
I20260810 18:51:58.651448 19307 leader_election.cc:288] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:58.654443 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:58.673503 16448 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:51:58.595960 19392 raft_consensus.cc:2781] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:51:58.577694 19198 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:58.587441 19366 raft_consensus.cc:3032] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 3 LEADER]: Stepping down as leader of term 3
W20260810 18:51:58.691495 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 167548us
I20260810 18:51:58.585639 19391 consensus_queue.cc:237] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:58.687306 19351 raft_consensus.cc:482] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:58.687718 18631 raft_consensus.cc:482] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:58.706869 16575 leader_election.cc:302] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:58.707464 16750 raft_consensus.cc:2379] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 2: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:51:58.722332 16448 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:51:58.709154 18942 raft_consensus.cc:482] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:58.494127 19418 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:58.706692 19198 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:51:58.701943 19366 raft_consensus.cc:729] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:58.712597 18728 raft_consensus.cc:2781] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:51:58.720846 19421 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:51:58.731344 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 391480us
I20260810 18:51:58.714927 19392 raft_consensus.cc:686] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:51:58.698781 18922 raft_consensus.cc:504] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:58.626370 19373 consensus_queue.cc:260] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:51:58.738271 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 327311us
I20260810 18:51:58.740074 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:51:58.718503 18957 consensus_meta.cc:287] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.528s user 0.005s sys 0.007s
I20260810 18:51:58.748346 16575 leader_election.cc:302] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:51:58.749521 17785 raft_consensus.cc:2454] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:51:58.763912 16695 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:51:58.750833 16575 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:51:58.750852 19162 raft_consensus.cc:686] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:51:58.723503 18631 raft_consensus.cc:504] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:58.733105 19304 consensus_queue.cc:1035] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260810 18:51:58.749050 19307 raft_consensus.cc:482] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:58.721287 19351 raft_consensus.cc:504] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:58.734907 18942 raft_consensus.cc:504] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:58.779338 19373 raft_consensus.cc:3037] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:58.777597 19421 raft_consensus.cc:482] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:58.804790 17781 raft_consensus.cc:1261] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260810 18:51:58.781189 18903 raft_consensus.cc:2781] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:51:58.807176 19429 raft_consensus.cc:482] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:58.756814 18957 raft_consensus.cc:504] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:58.790529 18730 raft_consensus.cc:2726] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:51:58.902235 16747 raft_consensus.cc:1261] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260810 18:51:58.903057 16747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1415) took 1249 ms. Trace:
I20260810 18:51:58.903801 16747 rpcz_store.cc:276] 0810 18:51:57.653248 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:58.900868 (+1247620us) service_pool.cc:227] Handling call
0810 18:51:58.901578 (+ 710us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:58.902763 (+ 1185us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:58.902959 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:51:58.831645 19382 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:58.914286 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:58.914484 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:51:58.928352 18957 leader_election.cc:288] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:58.929453 18957 raft_consensus.cc:2781] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:51:58.929971 18957 raft_consensus.cc:482] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:58.930529 18957 raft_consensus.cc:3037] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:51:58.905653 19168 raft_consensus.cc:482] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:58.933065 19168 raft_consensus.cc:504] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:58.937693 19168 leader_election.cc:288] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:58.938930 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:51:58.941944 16747 raft_consensus.cc:2379] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 4: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
W20260810 18:51:58.851346 19397 consensus_meta.cc:287] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.523s user 0.011s sys 0.003s
I20260810 18:51:58.843940 18903 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260810 18:51:58.907330 19339 consensus_meta.cc:287] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.659s user 0.009s sys 0.004s
I20260810 18:51:58.952663 16495 raft_consensus.cc:2454] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:51:58.959169 19029 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:51:58.964898 19441 log.cc:924] Time spent T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.249s user 0.000s sys 0.001s
I20260810 18:51:58.832484 19307 raft_consensus.cc:504] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:58.835568 19421 raft_consensus.cc:504] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:58.838876 19392 consensus_queue.cc:237] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:58.882500 19429 raft_consensus.cc:504] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:51:58.876688 19366 consensus_queue.cc:260] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:51:59.222873 16508 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.779s user 0.014s sys 0.096s
I20260810 18:51:59.007525 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:51:59.478233 19029 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:58.953699 19162 consensus_queue.cc:237] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:59.479543 16505 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56468 (request call id 129) took 1069 ms. Trace:
I20260810 18:51:59.480247 16505 rpcz_store.cc:276] 0810 18:51:58.409850 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:58.410056 (+ 206us) service_pool.cc:227] Handling call
0810 18:51:59.007317 (+597261us) spinlock_profiling.cc:243] Waited 2.92 ms on lock 0x7b5800001898. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed03c1b0 00007f1aed03ba94 00007f1ae6723c04 00007f1ae6723b92
0810 18:51:59.479477 (+472160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6723840}
W20260810 18:51:59.481678 16508 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.038s user 0.014s sys 0.142s
W20260810 18:51:59.482331 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 425292us
I20260810 18:51:59.484691 19429 leader_election.cc:288] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:59.489594 19429 raft_consensus.cc:2726] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:51:58.935102 19351 leader_election.cc:288] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:51:59.268618 16504 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52352 (request call id 1343) took 1601 ms (client timeout 1718 ms). Trace:
I20260810 18:51:59.496443 19351 raft_consensus.cc:2781] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:51:59.497186 19351 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:51:59.497947 19351 raft_consensus.cc:3037] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:51:59.500636 18942 leader_election.cc:288] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:59.503350 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:51:59.504273 17781 raft_consensus.cc:2454] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
W20260810 18:51:59.505604 19403 consensus_meta.cc:287] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 1.173s user 0.008s sys 0.005s
I20260810 18:51:59.506271 19403 raft_consensus.cc:2726] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260810 18:51:59.496738 16504 rpcz_store.cc:269] 0810 18:51:57.667167 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:57.667323 (+ 156us) service_pool.cc:227] Handling call
0810 18:51:59.268387 (+1601064us) spinlock_profiling.cc:243] Waited 95.7 ms on lock 0x7b54001cbbd4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:59.268542 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":220181888}
I20260810 18:51:59.507164 18975 raft_consensus.cc:2726] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:51:59.511613 19029 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:59.006073 19397 raft_consensus.cc:504] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:59.281980 18922 leader_election.cc:288] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:51:59.490513 17783 consensus_meta.cc:287] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.948s user 0.004s sys 0.009s
W20260810 18:51:59.048919 19439 log.cc:924] Time spent T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.374s user 0.002s sys 0.000s
I20260810 18:51:59.491993 19418 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:51:59.362355 19445 log.cc:924] Time spent T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.423s user 0.001s sys 0.000s
I20260810 18:51:58.904860 19340 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:51:59.220896 16448 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.567s (SENT) [suppressed 24 similar messages]
W20260810 18:51:59.292936 18827 consensus_meta.cc:287] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.910s user 0.010s sys 0.005s
I20260810 18:51:59.553388 19418 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:59.566656 18870 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:59.580008 18870 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:59.575800 19418 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:59.584532 19418 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:59.585474 19418 raft_consensus.cc:482] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:59.586259 19418 raft_consensus.cc:504] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:59.589143 19418 leader_election.cc:288] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:59.068111 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:59.004145 18903 raft_consensus.cc:3037] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:51:59.590963 16747 raft_consensus.cc:2454] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:51:59.596259 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:51:59.578533 16500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1313) took 1655 ms. Trace:
I20260810 18:51:59.598222 16500 rpcz_store.cc:276] 0810 18:51:57.923114 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:57.972116 (+ 49002us) service_pool.cc:227] Handling call
0810 18:51:58.023698 (+ 51582us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:58.543674 (+519976us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:59.578133 (+1034459us) spinlock_profiling.cc:243] Waited 473 ms on lock 0x7b54001ff9d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:51:59.578440 (+ 307us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1088872320}
I20260810 18:51:59.602295 16740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1424) took 1733 ms. Trace:
I20260810 18:51:59.603034 16740 rpcz_store.cc:276] 0810 18:51:57.868453 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:57.959917 (+ 91464us) service_pool.cc:227] Handling call
0810 18:51:57.960252 (+ 335us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:51:58.116194 (+155942us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:58.128761 (+ 12567us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:51:58.229931 (+101170us) log.cc:842] Serialized 26 byte log entry
0810 18:51:58.383679 (+153748us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:51:58.387704 (+ 4025us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:58.397229 (+ 9525us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:51:59.601909 (+1204680us) raft_consensus.cc:1671] finished
0810 18:51:59.601972 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:59.602203 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":69002,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":7443,"thread_start_us":145619,"threads_started":2,"wal-append.queue_time_us":157582}
I20260810 18:51:59.535573 17783 raft_consensus.cc:2454] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
W20260810 18:51:59.609724 17783 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55570 (request call id 366) took 1758 ms (client timeout 2151 ms). Trace:
W20260810 18:51:59.610482 17783 rpcz_store.cc:269] 0810 18:51:57.851529 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:57.857104 (+ 5575us) service_pool.cc:227] Handling call
0810 18:51:58.491683 (+634579us) spinlock_profiling.cc:243] Waited 4.33 ms on lock 0x7b5800005498. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed03c1b0 00007f1aed03ba94 00007f1ae6723c04 00007f1ae6723b92
0810 18:51:59.609613 (+1117930us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9953536}
I20260810 18:51:59.613586 18870 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:51:59.622010 19373 consensus_meta.cc:287] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.797s user 0.006s sys 0.006s
W20260810 18:51:59.622929 19435 consensus_meta.cc:287] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 1.116s user 0.010s sys 0.006s
I20260810 18:51:59.624032 16740 raft_consensus.cc:1226] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260810 18:51:59.624343 16752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:45870 (request call id 6) took 1745 ms. Trace:
W20260810 18:51:59.555681 19365 consensus_meta.cc:287] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 1.167s user 0.009s sys 0.008s
I20260810 18:51:59.777071 19365 raft_consensus.cc:504] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:59.573030 19397 leader_election.cc:288] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:59.627213 19135 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:51:59.649109 19396 log.cc:924] Time spent T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 1.686s user 0.000s sys 0.011s
I20260810 18:51:59.669219 16742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37716 (request call id 125) took 1252 ms. Trace:
I20260810 18:51:59.676358 16745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1427) took 1649 ms. Trace:
I20260810 18:51:59.678798 16750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37716 (request call id 128) took 1157 ms. Trace:
I20260810 18:51:59.204136 19393 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:59.062101 18631 leader_election.cc:288] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:59.684494 19354 raft_consensus.cc:482] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:59.698323 16507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1360) took 1360 ms. Trace:
I20260810 18:51:59.805809 16507 rpcz_store.cc:276] 0810 18:51:58.338069 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:58.338211 (+ 142us) service_pool.cc:227] Handling call
0810 18:51:58.338602 (+ 391us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:58.517284 (+178682us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:58.517314 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:51:58.517362 (+ 48us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:51:58.517391 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:59.698020 (+1180629us) spinlock_profiling.cc:243] Waited 178 ms on lock 0x7b54001374d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:51:59.698074 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:59.698247 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":410447616}
I20260810 18:51:59.796681 19135 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:59.807147 19114 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:59.807123 19354 raft_consensus.cc:504] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:59.807946 19365 leader_election.cc:288] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:59.807796 19114 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:59.808949 19135 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:59.809422 19354 leader_election.cc:288] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:58.955614 19382 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:59.641873 18827 raft_consensus.cc:504] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:59.758038 19406 raft_consensus.cc:2229] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Raft consensus shutting down.
I20260810 18:51:59.022043 18931 consensus_queue.cc:1035] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.690s
I20260810 18:51:59.015865 19339 raft_consensus.cc:504] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:59.776041 16752 rpcz_store.cc:276] 0810 18:51:57.879233 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:57.914880 (+ 35647us) service_pool.cc:227] Handling call
0810 18:51:59.624066 (+1709186us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b540020a654. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f4bc3 00007f1aed03e81a 00007f1ae6731d04 00007f1ae6731c92 00007f1ae6731bc1 00007f1ae6731b3c 00007f1ae6730df2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:59.624194 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":415447936}
I20260810 18:51:59.791217 16750 rpcz_store.cc:276] 0810 18:51:58.521730 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:58.522411 (+ 681us) service_pool.cc:227] Handling call
0810 18:51:59.678539 (+1156128us) spinlock_profiling.cc:243] Waited 153 ms on lock 0x7b540034e9d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:59.678723 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":352076032}
I20260810 18:51:59.789775 16745 rpcz_store.cc:276] 0810 18:51:58.027350 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:58.027543 (+ 193us) service_pool.cc:227] Handling call
0810 18:51:58.027948 (+ 405us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:51:58.136016 (+108068us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:58.136045 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:51:58.176147 (+ 40102us) log.cc:842] Serialized 26 byte log entry
0810 18:51:59.341852 (+1165705us) spinlock_profiling.cc:243] Waited 3.46 ms on lock 0x7b18003e4100. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1aee0718b1 00007f1aee07e661 00007f1ae7d8982c 00007f1ae51b49dc 00007f1aedfeba39 00007f1ae51d18bb 00007f1ae51cfcbf 00007f1ae51d1e22 00007f1ae7d7b5bb 00007f1ae7d81063 00007f1ae7d81411
0810 18:51:59.342068 (+ 216us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:51:59.342117 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:59.675962 (+333845us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b540020a654. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:51:59.676020 (+ 58us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:51:59.676087 (+ 67us) raft_consensus.cc:1671] finished
0810 18:51:59.676105 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:59.676285 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3422,"raft.queue_time_us":136512,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":15022,"spinlock_wait_cycles":255575168,"thread_start_us":47810,"threads_started":1,"wal-append.queue_time_us":111290,"wal-append.run_cpu_time_us":40086,"wal-append.run_wall_time_us":1158708}
I20260810 18:51:59.809906 19114 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:59.818066 16740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1440) took 1063 ms. Trace:
I20260810 18:51:59.842818 19406 raft_consensus.cc:2258] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Raft consensus is shut down!
I20260810 18:51:59.844872 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:51:59.822150 19156 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.379s user 0.049s sys 0.122s
I20260810 18:51:59.857414 16752 raft_consensus.cc:2454] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 4.
W20260810 18:51:59.864274 19156 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.442s user 0.050s sys 0.122s
I20260810 18:51:59.790831 16742 rpcz_store.cc:276] 0810 18:51:58.416581 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:58.507981 (+ 91400us) service_pool.cc:227] Handling call
0810 18:51:59.668946 (+1160965us) spinlock_profiling.cc:243] Waited 37.3 ms on lock 0x7b540029d954. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:51:59.669137 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":85746048}
I20260810 18:51:59.778590 19435 raft_consensus.cc:504] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:51:59.883330 18957 consensus_meta.cc:287] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.952s user 0.015s sys 0.000s
W20260810 18:51:59.885598 19198 consensus_meta.cc:287] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 1.146s user 0.012s sys 0.003s
I20260810 18:51:59.860441 17775 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55570 (request call id 370) took 1565 ms. Trace:
W20260810 18:51:59.901641 16574 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.624s user 0.019s sys 0.058s
W20260810 18:51:59.915618 16574 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.775s user 0.019s sys 0.058s
I20260810 18:51:59.917349 16749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 119) took 2096 ms. Trace:
I20260810 18:51:59.897862 19198 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:59.919836 16749 rpcz_store.cc:276] 0810 18:51:57.821432 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:57.939668 (+118236us) service_pool.cc:227] Handling call
0810 18:51:57.963963 (+ 24295us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:51:58.285222 (+321259us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:58.285271 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:51:58.309827 (+ 24556us) log.cc:842] Serialized 26 byte log entry
0810 18:51:58.714983 (+405156us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:51:58.715032 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:58.715142 (+ 110us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:51:59.917059 (+1201917us) raft_consensus.cc:1671] finished
0810 18:51:59.917091 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:59.917278 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":104912,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":5144,"thread_start_us":618723,"threads_started":2,"wal-append.queue_time_us":1399803}
I20260810 18:51:59.919270 18903 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:59.896672 17775 rpcz_store.cc:276] 0810 18:51:58.275424 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:58.276529 (+ 1105us) service_pool.cc:227] Handling call
0810 18:51:58.318888 (+ 42359us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:51:58.326283 (+ 7395us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:58.326331 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:51:58.326903 (+ 572us) log.cc:842] Serialized 26 byte log entry
0810 18:51:58.457593 (+130690us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:51:58.457657 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:58.911620 (+453963us) spinlock_profiling.cc:243] Waited 6.83 ms on lock 0x7b54006a1654. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:51:58.919309 (+ 7689us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:51:59.789945 (+870636us) raft_consensus.cc:1671] finished
0810 18:51:59.790795 (+ 850us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:51:59.840586 (+ 49791us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5603,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":6186,"spinlock_wait_cycles":15703296,"thread_start_us":124085,"threads_started":1,"wal-append.queue_time_us":453229}
I20260810 18:51:59.926726 19397 raft_consensus.cc:482] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:59.927413 19432 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:51:59.927254 19397 raft_consensus.cc:504] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:59.927906 19432 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:51:59.929275 19397 leader_election.cc:288] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:59.929847 19432 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:59.930774 19365 raft_consensus.cc:482] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:51:59.931592 19435 leader_election.cc:288] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:51:59.931296 19365 raft_consensus.cc:504] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:59.845736 16740 rpcz_store.cc:276] 0810 18:51:58.747493 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:59.606716 (+859223us) service_pool.cc:227] Handling call
0810 18:51:59.623755 (+ 17039us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:51:59.787736 (+163981us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:59.810705 (+ 22969us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:51:59.895795 19421 leader_election.cc:288] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:51:59.937443 19365 leader_election.cc:288] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:59.913054 19307 leader_election.cc:288] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:51:59.896823 18957 raft_consensus.cc:504] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:51:59.938181 19435 raft_consensus.cc:482] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:51:59.976507 16447 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.233s user 0.050s sys 0.061s
W20260810 18:51:59.977968 16447 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.234s user 0.050s sys 0.061s
I20260810 18:51:59.960402 17727 leader_election.cc:302] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:00.072764 19172 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:00.073545 19367 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:00.073426 19172 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:00.074206 19367 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:00.080648 19448 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:00.081278 19448 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:00.084168 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:00.085007 16500 raft_consensus.cc:2454] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 3.
I20260810 18:52:00.091001 19406 ts_tablet_manager.cc:1905] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 18:52:00.033105 18827 leader_election.cc:288] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:00.033526 18903 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:00.097779 16508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56468 (request call id 127) took 1854 ms. Trace:
I20260810 18:52:00.098249 16508 rpcz_store.cc:276] 0810 18:51:58.243409 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:58.243927 (+ 518us) service_pool.cc:227] Handling call
0810 18:51:58.267198 (+ 23271us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:51:58.332534 (+ 65336us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:58.332568 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:51:58.390040 (+ 57472us) log.cc:842] Serialized 26 byte log entry
0810 18:51:59.482883 (+1092843us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:51:59.482934 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:59.624829 (+141895us) spinlock_profiling.cc:243] Waited 65 ms on lock 0x7b54001392d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:51:59.624891 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:00.097366 (+472475us) raft_consensus.cc:1671] finished
0810 18:52:00.097408 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:00.097693 (+ 285us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":51261,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":11569,"spinlock_wait_cycles":149478144,"thread_start_us":1039392,"threads_started":1,"wal-append.queue_time_us":1656476}
I20260810 18:51:59.975917 19435 raft_consensus.cc:504] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:00.076159 19339 leader_election.cc:288] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:00.067086 19388 raft_consensus.cc:482] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:00.070304 19382 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:00.114398 19380 raft_consensus.cc:2781] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260810 18:52:00.112852 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 159485us
W20260810 18:52:00.158787 16450 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.346s user 0.006s sys 0.072s
I20260810 18:52:00.160327 19339 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:00.162142 16450 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.349s user 0.006s sys 0.073s
I20260810 18:52:00.117621 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:52:00.163542 19291 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.385s user 0.006s sys 0.022s
W20260810 18:52:00.169682 19291 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.394s user 0.007s sys 0.022s
I20260810 18:52:00.169368 19339 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:00.158746 19380 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260810 18:52:00.154615 18728 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.378s user 0.035s sys 0.044s
W20260810 18:52:00.161082 16575 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.396s user 0.016s sys 0.040s
W20260810 18:52:00.263695 16575 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.498s user 0.017s sys 0.040s
I20260810 18:52:00.264397 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:00.266122 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 1.51523e+06us
I20260810 18:52:00.268239 16575 leader_election.cc:302] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:00.267086 16504 raft_consensus.cc:2454] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:52:00.271582 16450 leader_election.cc:302] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:00.168883 19198 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:00.126436 19388 raft_consensus.cc:504] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:00.283281 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:00.260376 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:52:00.266942 18728 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.494s user 0.036s sys 0.045s
I20260810 18:52:00.289168 16746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 123) took 2011 ms. Trace:
I20260810 18:52:00.299649 16505 raft_consensus.cc:1261] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260810 18:52:00.264897 19380 raft_consensus.cc:3037] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:00.266700 19448 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:00.275552 19307 raft_consensus.cc:2781] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:52:00.298055 16746 rpcz_store.cc:276] 0810 18:51:58.277982 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:58.278447 (+ 465us) service_pool.cc:227] Handling call
0810 18:51:58.278771 (+ 324us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:51:58.317780 (+ 39009us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:58.317810 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:51:58.318278 (+ 468us) log.cc:842] Serialized 26 byte log entry
0810 18:51:58.394370 (+ 76092us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:51:58.394412 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:51:58.394482 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:00.283894 (+1889412us) spinlock_profiling.cc:243] Waited 66.1 ms on lock 0x7b24001c1828. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae502acf4 00007f1ae5028dfb 00007f1ae50b7559 00007f1ae50b71e5 00007f1ae50b6740 00007f1ae7da9578 00007f1ae7da9cd8 00007f1ae7d816e8 00007f1ae7d7caea 00007f1ae7d7bf36
0810 18:52:00.284898 (+ 1004us) raft_consensus.cc:1671] finished
0810 18:52:00.284925 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:00.289087 (+ 4162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35372,"raft.queue_time_us":39497,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":97,"spinlock_wait_cycles":152103168,"thread_start_us":75684,"threads_started":1,"wal-append.queue_time_us":226905}
W20260810 18:52:00.311277 16701 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.479s user 0.005s sys 0.028s
W20260810 18:52:00.313982 16701 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.482s user 0.005s sys 0.028s
I20260810 18:52:00.285815 19304 raft_consensus.cc:2781] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:00.321658 19397 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:00.285126 19366 raft_consensus.cc:3037] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
W20260810 18:52:00.298200 19099 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.685s user 0.048s sys 0.061s
W20260810 18:52:00.311709 18808 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.351s user 0.009s sys 0.031s
I20260810 18:52:00.325417 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:00.327195 16623 raft_consensus.cc:1226] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3. Ops: []
W20260810 18:52:00.336470 16573 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.636s user 0.006s sys 0.036s
W20260810 18:52:00.337996 16573 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.637s user 0.006s sys 0.036s
W20260810 18:52:00.306034 19312 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.806s user 0.011s sys 0.023s
W20260810 18:52:00.328106 16748 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.578s user 0.013s sys 0.027s
W20260810 18:52:00.322250 19385 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.867s user 0.109s sys 0.219s
I20260810 18:52:00.279325 19365 consensus_queue.cc:1046] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260810 18:52:00.318888 19432 raft_consensus.cc:482] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:00.338708 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:00.339011 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:00.343900 16636 raft_consensus.cc:2454] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
W20260810 18:52:00.340166 16748 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.590s user 0.013s sys 0.027s
I20260810 18:52:00.340456 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:00.340749 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:00.345538 16626 raft_consensus.cc:2454] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
I20260810 18:52:00.327909 19307 raft_consensus.cc:686] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
W20260810 18:52:00.334348 19099 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.716s user 0.049s sys 0.061s
I20260810 18:52:00.310256 18728 raft_consensus.cc:686] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:00.350430 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:00.351262 16622 raft_consensus.cc:2379] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 5: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:00.345796 16624 raft_consensus.cc:2396] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20260810 18:52:00.336484 18808 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.374s user 0.010s sys 0.031s
I20260810 18:52:00.332032 19304 raft_consensus.cc:686] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:00.340423 17783 raft_consensus.cc:2396] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260810 18:52:00.340483 19385 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.891s user 0.110s sys 0.220s
W20260810 18:52:00.342346 16743 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.548s user 0.011s sys 0.031s
W20260810 18:52:00.356509 16743 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.562s user 0.011s sys 0.031s
I20260810 18:52:00.346727 16500 raft_consensus.cc:1261] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260810 18:52:00.344173 19367 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:00.351056 19190 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.792s user 0.015s sys 0.028s
I20260810 18:52:00.332227 19397 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:00.352035 19435 leader_election.cc:288] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:00.352528 19198 raft_consensus.cc:3032] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:52:00.353467 19172 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:00.366668 19198 raft_consensus.cc:729] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:00.370325 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:00.370932 19388 leader_election.cc:288] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:00.371151 16624 raft_consensus.cc:2379] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 5: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:00.371049 19198 consensus_queue.cc:260] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:00.371862 19367 consensus_queue.cc:1035] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:52:00.373170 19307 consensus_queue.cc:237] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:00.361706 17780 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.526s user 0.049s sys 0.064s
W20260810 18:52:00.377230 17780 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.547s user 0.050s sys 0.064s
I20260810 18:52:00.377485 19339 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:00.348474 19432 raft_consensus.cc:504] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:00.372908 19198 raft_consensus.cc:3037] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:00.379551 19432 leader_election.cc:288] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:00.380609 19304 consensus_queue.cc:237] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:00.341420 19312 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.843s user 0.011s sys 0.024s
I20260810 18:52:00.356922 16605 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ef7c65533724e678b75830002906e2c with delete_type TABLET_DATA_TOMBSTONED (TS 0397d760589d40d3b92ef141cdfc2dbb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53868
I20260810 18:52:00.384400 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:00.385032 16624 raft_consensus.cc:2454] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
W20260810 18:52:00.350948 19168 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.385s user 0.026s sys 0.015s
W20260810 18:52:00.386188 19168 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.420s user 0.027s sys 0.015s
I20260810 18:52:00.387432 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:00.388025 16624 raft_consensus.cc:2454] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
W20260810 18:52:00.389153 19419 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.048s user 0.000s sys 0.015s
W20260810 18:52:00.389467 19419 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.097s user 0.000s sys 0.015s
W20260810 18:52:00.390085 19391 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.874s user 0.000s sys 0.015s
I20260810 18:52:00.391546 16631 raft_consensus.cc:1261] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260810 18:52:00.392491 19397 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:00.365463 19190 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.807s user 0.015s sys 0.028s
W20260810 18:52:00.367911 16701 connection.cc:232] RPC call timeout handler was delayed by -1.4456563659477979s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2182s for a response
W20260810 18:52:00.401150 16702 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.842s user 0.005s sys 0.011s
W20260810 18:52:00.401577 16702 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.847s user 0.005s sys 0.011s
I20260810 18:52:00.398692 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:00.395387 19338 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.556s user 0.061s sys 0.086s
I20260810 18:52:00.403370 16624 raft_consensus.cc:2454] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
W20260810 18:52:00.399557 16572 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.628s user 0.010s sys 0.036s
W20260810 18:52:00.404893 16572 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.718s user 0.011s sys 0.036s
I20260810 18:52:00.403714 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:00.399286 19391 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.883s user 0.000s sys 0.015s
W20260810 18:52:00.406131 19338 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.571s user 0.061s sys 0.086s
W20260810 18:52:00.498052 17726 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.386s user 0.002s sys 0.015s
I20260810 18:52:00.402542 16495 raft_consensus.cc:1226] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:00.402681 18808 raft_consensus.cc:482] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:00.405706 19382 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:00.560637 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:00.570537 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:00.570590 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:00.571333 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:00.655599 16505 raft_consensus.cc:3037] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:00.572830 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:00.580120 19391 raft_consensus.cc:482] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:00.583004 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:00.584551 19190 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:00.589489 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:00.600486 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:00.590209 16496 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.933s user 0.005s sys 0.018s
W20260810 18:52:00.593659 18948 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.768s user 0.006s sys 0.019s
I20260810 18:52:00.589507 16500 raft_consensus.cc:1226] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 2. Current term is 3. Ops: []
I20260810 18:52:00.671002 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:00.593909 19156 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.696s user 0.014s sys 0.005s
I20260810 18:52:00.443894 18728 consensus_queue.cc:237] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:00.598430 19162 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.781s user 0.008s sys 0.012s
W20260810 18:52:00.639549 17726 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.528s user 0.003s sys 0.015s
W20260810 18:52:00.670380 16443 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.137s user 0.003s sys 0.012s
W20260810 18:52:00.662092 19289 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.072s user 0.028s sys 0.021s
W20260810 18:52:00.637037 19351 consensus_meta.cc:287] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 1.138s user 0.011s sys 0.006s
I20260810 18:52:00.654326 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:00.669617 16496 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.013s user 0.006s sys 0.018s
I20260810 18:52:00.665146 19391 raft_consensus.cc:504] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:00.672119 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:00.674017 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:00.651818 19323 raft_consensus.cc:482] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:00.652045 16622 raft_consensus.cc:2454] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:00.681504 16624 raft_consensus.cc:2454] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:00.659373 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:00.692754 16508 raft_consensus.cc:3037] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
W20260810 18:52:00.669006 18948 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.853s user 0.006s sys 0.019s
W20260810 18:52:00.674492 19156 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.777s user 0.014s sys 0.006s
I20260810 18:52:00.697805 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:00.677613 19162 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.857s user 0.008s sys 0.012s
W20260810 18:52:00.676211 19289 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.087s user 0.028s sys 0.021s
W20260810 18:52:00.682416 19163 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.923s user 0.004s sys 0.025s
W20260810 18:52:00.674646 16443 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.404s user 0.004s sys 0.013s
I20260810 18:52:00.659660 18808 raft_consensus.cc:504] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:00.649613 18903 raft_consensus.cc:482] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:00.688050 19077 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.154s user 0.007s sys 0.028s
W20260810 18:52:00.691358 19279 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.056s user 0.002s sys 0.032s
I20260810 18:52:00.660820 19382 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:00.662225 19380 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:00.703845 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:52:00.705490 19163 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.991s user 0.004s sys 0.025s
I20260810 18:52:00.703611 16502 raft_consensus.cc:1226] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3. Ops: []
I20260810 18:52:00.707880 19496 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:00.723351 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:00.709669 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:00.716748 19279 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.084s user 0.003s sys 0.033s
I20260810 18:52:00.671571 19190 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:00.716005 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 433538us
I20260810 18:52:00.728663 16450 leader_election.cc:302] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:52:00.715605 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 438109us
W20260810 18:52:00.720081 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 319519us
I20260810 18:52:00.683497 19351 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:00.714000 19077 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.182s user 0.007s sys 0.028s
W20260810 18:52:00.695807 16695 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.142s user 0.016s sys 0.021s
W20260810 18:52:00.733831 16695 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.180s user 0.016s sys 0.021s
W20260810 18:52:00.724169 19460 log.cc:924] Time spent T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.085s user 0.000s sys 0.001s
W20260810 18:52:00.718241 19493 log.cc:924] Time spent T 00000000000000000000000000000000 P 0975c6fa18274f2baee7fb3615d5b985: Append to log took a long time: real 0.059s user 0.001s sys 0.001s
I20260810 18:52:00.729795 17724 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:00.730688 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:00.711259 16506 raft_consensus.cc:1226] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 2. Current term is 3. Ops: []
I20260810 18:52:00.735733 17778 raft_consensus.cc:1226] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:00.726142 19496 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:00.736142 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:00.736729 16622 raft_consensus.cc:3037] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:00.717217 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:00.737799 16701 connection.cc:232] RPC call timeout handler was delayed by -1.3434637440368533s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1877s for a response
W20260810 18:52:00.719631 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 438878us
I20260810 18:52:00.713467 19323 raft_consensus.cc:504] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:00.716885 18903 raft_consensus.cc:504] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:00.765403 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:00.795974 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:01.058418 16628 raft_consensus.cc:2454] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:00.797327 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:00.796506 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:00.909420 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 180858us
W20260810 18:52:00.765522 16701 connection.cc:232] RPC call timeout handler was delayed by -1.3234680460300297s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16770000000000002s for a response
W20260810 18:52:00.852376 16507 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.577s user 0.014s sys 0.003s
W20260810 18:52:01.070217 16507 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.795s user 0.024s sys 0.006s
I20260810 18:52:01.056861 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:00.795132 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:00.852059 16748 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.014s sys 0.003s
W20260810 18:52:00.929351 16605 thread.cc:636] tablet-delete [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.002s sys 0.037s
W20260810 18:52:01.101708 16748 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.754s user 0.014s sys 0.017s
I20260810 18:52:00.797703 16505 raft_consensus.cc:2454] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:00.821355 16622 raft_consensus.cc:1261] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260810 18:52:01.093889 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:01.098991 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:01.103488 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:01.104386 16633 raft_consensus.cc:2379] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 3: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:01.073096 16504 raft_consensus.cc:2379] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 3: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:01.071506 16450 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:01.104460 16628 raft_consensus.cc:2379] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 3: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:01.100767 16695 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
W20260810 18:52:01.107028 19510 log.cc:924] Time spent T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.117s user 0.000s sys 0.000s
I20260810 18:52:00.785995 16496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56468 (request call id 130) took 2365 ms. Trace:
I20260810 18:52:00.791720 17775 raft_consensus.cc:1261] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260810 18:52:01.106961 16495 raft_consensus.cc:3037] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:52:00.939409 19477 raft_consensus.cc:482] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:00.929574 19475 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:00.940224 19448 raft_consensus.cc:2726] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260810 18:52:00.938887 17780 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.007s sys 0.033s
I20260810 18:52:00.834462 19391 leader_election.cc:288] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:01.110105 16626 raft_consensus.cc:2454] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:01.105744 19496 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:01.111109 16748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1436) took 2384 ms. Trace:
I20260810 18:52:01.109957 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:01.142854 16628 raft_consensus.cc:2379] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 5: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:01.140213 16748 rpcz_store.cc:276] 0810 18:51:58.718055 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:58.718245 (+ 190us) service_pool.cc:227] Handling call
0810 18:51:58.732089 (+ 13844us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:58.732331 (+ 242us) raft_consensus.cc:1520] Early marking committed up to index 2
0810 18:52:00.347406 (+1615075us) log.cc:842] Serialized 14 byte log entry
0810 18:52:01.102134 (+754728us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:01.102184 (+ 50us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:01.102229 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:01.102297 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:01.102418 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1839688,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":264,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":9610,"thread_start_us":2350742,"threads_started":2}
I20260810 18:52:01.144023 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:01.114087 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:01.114814 19380 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:01.114972 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:01.115496 19190 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:01.115875 19382 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:01.116420 19366 consensus_meta.cc:287] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.782s user 0.006s sys 0.006s
I20260810 18:52:01.116732 18957 leader_election.cc:288] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:01.107870 19473 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:01.118546 19470 consensus_queue.cc:1046] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:52:01.154898 19323 leader_election.cc:288] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:01.118924 19504 raft_consensus.cc:2781] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:52:01.119550 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:01.100054 19462 consensus_queue.cc:1046] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260810 18:52:01.159878 19323 raft_consensus.cc:3032] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:52:01.115084 18808 leader_election.cc:288] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:01.160822 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:01.161655 16624 raft_consensus.cc:2354] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for earlier term 2. Current term is 3.
I20260810 18:52:01.162384 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:01.121814 16448 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.557s user 0.005s sys 0.014s
W20260810 18:52:01.163250 16448 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.599s user 0.005s sys 0.015s
W20260810 18:52:01.163692 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 1.9427e+06us
W20260810 18:52:01.107903 16605 thread.cc:602] tablet-delete [worker] (thread pool) Time spent starting thread: real 0.724s user 0.002s sys 0.037s
I20260810 18:52:01.126073 19156 raft_consensus.cc:2781] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:01.111123 17775 pending_rounds.cc:76] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd: Aborting all ops after (but not including) 4
I20260810 18:52:01.180017 19366 raft_consensus.cc:3032] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 LEADER]: Stepping down as leader of term 4
I20260810 18:52:00.794072 19351 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:01.132534 16508 raft_consensus.cc:2454] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
I20260810 18:52:00.929446 19495 raft_consensus.cc:482] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:01.133687 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:01.135181 16633 raft_consensus.cc:1226] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260810 18:52:00.754153 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
W20260810 18:52:01.141083 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 411362us
I20260810 18:52:01.145766 16632 raft_consensus.cc:2454] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:01.146747 16628 raft_consensus.cc:2379] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 4: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
W20260810 18:52:01.135334 17780 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.757s user 0.008s sys 0.033s
I20260810 18:52:01.152655 16636 raft_consensus.cc:2454] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:01.120124 19406 ts_tablet_manager.cc:1918] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260810 18:52:01.154271 19473 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:01.155568 16743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 124) took 2854 ms. Trace:
I20260810 18:52:01.134981 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:01.160328 19323 raft_consensus.cc:729] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:01.163190 16506 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1315) took 2777 ms. Trace:
I20260810 18:52:01.254212 16743 rpcz_store.cc:276] 0810 18:51:58.301510 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:58.316590 (+ 15080us) service_pool.cc:227] Handling call
0810 18:51:58.316877 (+ 287us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:51:58.752188 (+435311us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:58.752227 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:51:58.752705 (+ 478us) log.cc:842] Serialized 26 byte log entry
0810 18:52:00.358066 (+1605361us) raft_consensus.cc:1631] Marking committed up to 6
0810 18:52:00.358103 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:01.148836 (+790733us) spinlock_profiling.cc:243] Waited 428 ms on lock 0x7b54000708d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:01.148907 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:01.148977 (+ 70us) raft_consensus.cc:1671] finished
0810 18:52:01.148995 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:01.155479 (+ 6484us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7204,"raft.queue_time_us":10392,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":10643,"spinlock_wait_cycles":985546624,"thread_start_us":1563944,"threads_started":1,"wal-append.queue_time_us":1831896}
I20260810 18:52:01.108084 19099 raft_consensus.cc:482] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:01.169086 16623 raft_consensus.cc:1226] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 2. Current term is 3. Ops: []
I20260810 18:52:01.256881 16506 rpcz_store.cc:276] 0810 18:51:58.385654 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:58.403157 (+ 17503us) service_pool.cc:227] Handling call
0810 18:51:59.364771 (+961614us) spinlock_profiling.cc:243] Waited 3.16 ms on lock 0x7b5800001898. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519ab7c 00007f1ae519a73b 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed0502b0 00007f1aed03b7f9 00007f1aed03b2a2 00007f1ae6721684
0810 18:51:59.365036 (+ 265us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:00.744677 (+1379641us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:01.162961 (+418284us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b5400285854. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:01.163133 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3102258688}
I20260810 18:52:01.168998 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:01.162961 16624 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 4.
I20260810 18:52:01.158619 19504 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:01.107658 16496 rpcz_store.cc:276] 0810 18:51:58.420849 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:58.522252 (+101403us) service_pool.cc:227] Handling call
0810 18:51:58.654068 (+131816us) raft_consensus.cc:1457] Updating replica for 2 ops
0810 18:51:58.654845 (+ 777us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:51:58.654918 (+ 73us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0810 18:51:58.656317 (+ 1399us) log.cc:842] Serialized 52 byte log entry
0810 18:52:00.691989 (+2035672us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:00.697390 (+ 5401us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:00.702769 (+ 5379us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:00.785730 (+ 82961us) raft_consensus.cc:1671] finished
0810 18:52:00.785757 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:00.785910 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6368,"raft.queue_time_us":5293,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":27811,"thread_start_us":2027589,"threads_started":1,"wal-append.queue_time_us":2127589}
I20260810 18:52:01.179916 19515 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:01.176474 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:01.151244 19519 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
W20260810 18:52:01.188633 16448 connection.cc:232] RPC call timeout handler was delayed by -1.8925287709571421s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18000000000000002s for a response
I20260810 18:52:01.196566 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:01.198024 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:01.268195 16622 raft_consensus.cc:2379] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 4: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:01.181118 19156 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:01.197189 19495 raft_consensus.cc:504] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:01.211771 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:01.272876 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:01.239010 19291 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:01.180980 17775 pending_rounds.cc:97] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260810 18:52:01.187867 19366 raft_consensus.cc:729] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:01.263186 19504 raft_consensus.cc:3037] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:01.131968 19477 raft_consensus.cc:504] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:01.259189 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:01.259213 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:01.264954 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:01.266043 16746 raft_consensus.cc:2396] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260810 18:52:01.272464 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:01.292155 16752 raft_consensus.cc:3037] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:01.292445 16624 raft_consensus.cc:3037] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:01.133390 19475 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:01.258116 19524 tablet_replica.cc:330] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb: stopping tablet replica
I20260810 18:52:01.276130 16633 raft_consensus.cc:2454] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 4.
I20260810 18:52:01.295154 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:01.239547 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:01.259151 19391 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:01.301998 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:01.280138 16626 raft_consensus.cc:1261] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260810 18:52:01.254040 19351 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:01.281965 16448 connection.cc:232] RPC call timeout handler was delayed by -1.4094437551684678s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20910000000000004s for a response
I20260810 18:52:01.238960 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:01.322744 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:01.323575 16504 raft_consensus.cc:2379] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 5: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:01.261266 16742 raft_consensus.cc:3032] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:52:01.284513 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:01.325857 16740 raft_consensus.cc:2454] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:52:01.326097 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:01.247288 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:01.326364 16743 raft_consensus.cc:2454] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:52:01.274178 19406 log.cc:1196] T 03cb267b222b440981c94d431e7273bf P d304ac47fdf846de81337057a4ef7bea: Deleting WAL directory at /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-0-root/wals/03cb267b222b440981c94d431e7273bf
I20260810 18:52:01.326972 16505 raft_consensus.cc:2379] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 3: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:01.327414 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:01.327482 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:01.328253 19406 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52330 (request call id 791) took 3644 ms. Trace:
I20260810 18:52:01.293545 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:01.328959 19475 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:01.291260 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:01.329246 16749 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:01.272972 19373 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:01.329795 16622 raft_consensus.cc:1261] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260810 18:52:01.273557 19156 raft_consensus.cc:3037] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:01.309391 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:01.331979 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:01.310093 16448 connection.cc:232] RPC call timeout handler was delayed by -1.3588024519849569s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21810000000000002s for a response
W20260810 18:52:01.335948 16448 connection.cc:232] RPC call timeout handler was delayed by -1.2911697889212519s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20040000000000002s for a response
W20260810 18:52:01.336366 16448 connection.cc:232] RPC call timeout handler was delayed by -1.2711139880120754s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15580000000000002s for a response
W20260810 18:52:01.336716 16448 connection.cc:232] RPC call timeout handler was delayed by -1.0755502867978066s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2614s for a response
W20260810 18:52:01.337474 16448 leader_election.cc:334] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.558s (SENT)
I20260810 18:52:01.300912 16628 raft_consensus.cc:1261] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260810 18:52:01.284937 17775 raft_consensus.cc:2866] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:52:01.326148 16742 raft_consensus.cc:729] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:01.295776 16623 raft_consensus.cc:2454] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:52:01.321022 16752 raft_consensus.cc:2454] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:01.332609 16504 raft_consensus.cc:2454] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
I20260810 18:52:01.341713 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:01.263401 19099 raft_consensus.cc:504] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:01.345146 17726 connection.cc:232] RPC call timeout handler was delayed by -1.1691286559216678s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1986s for a response
I20260810 18:52:01.342373 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:01.294188 18728 raft_consensus.cc:482] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:01.294885 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:01.302834 19480 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:01.297465 17780 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55586 (request call id 270) took 2627 ms. Trace:
I20260810 18:52:01.297091 18808 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:01.273717 19515 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:01.355852 16623 raft_consensus.cc:1226] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 1. Current term is 2. Ops: []
I20260810 18:52:01.354336 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:01.357988 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 249305us
I20260810 18:52:01.359141 16507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1363) took 1097 ms. Trace:
I20260810 18:52:01.368324 16752 raft_consensus.cc:2454] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:01.286487 19291 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:01.329312 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:01.345679 17726 connection.cc:232] RPC call timeout handler was delayed by -1.0548286559060216s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21130000000000002s for a response
I20260810 18:52:01.334931 18903 leader_election.cc:288] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:01.341041 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:01.371560 16749 raft_consensus.cc:1261] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:52:01.368247 16507 rpcz_store.cc:276] 0810 18:52:00.261823 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:00.262072 (+ 249us) service_pool.cc:227] Handling call
0810 18:52:00.262419 (+ 347us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:00.262655 (+ 236us) raft_consensus.cc:1520] Early marking committed up to index 2
0810 18:52:01.070619 (+807964us) log.cc:842] Serialized 14 byte log entry
0810 18:52:01.358805 (+288186us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:01.358859 (+ 54us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:01.358895 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:01.358970 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:01.359087 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":94094,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":979,"thread_start_us":1053243,"threads_started":2}
I20260810 18:52:01.353607 19163 raft_consensus.cc:482] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:01.374305 16701 connection.cc:232] RPC call timeout handler was delayed by -1.0556664799805731s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16920000000000002s for a response
I20260810 18:52:01.384114 16752 raft_consensus.cc:1261] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260810 18:52:01.330097 19373 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:01.391980 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:01.361245 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:01.394268 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:01.281016 19519 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:01.393849 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:01.391350 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:01.362989 19029 raft_consensus.cc:2781] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:01.303259 19323 consensus_queue.cc:260] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:01.399565 16573 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:01.330631 19391 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:01.372188 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:01.330427 19351 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:01.355937 18728 raft_consensus.cc:504] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:01.352869 19366 consensus_queue.cc:260] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:01.370163 18808 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:01.361605 19480 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:01.421008 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
W20260810 18:52:01.556361 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 217951us
I20260810 18:52:01.391593 18903 raft_consensus.cc:482] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:01.530789 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:52:01.603312 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 268592us
I20260810 18:52:01.392508 17770 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:01.610049 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:01.412703 16752 raft_consensus.cc:2379] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 5: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
W20260810 18:52:01.413720 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 312997us
I20260810 18:52:01.508497 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:01.328655 19406 rpcz_store.cc:276] 0810 18:51:57.683465 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:57.683668 (+ 203us) service_pool.cc:227] Handling call
0810 18:51:58.314217 (+630549us) ts_tablet_manager.cc:1145] Deleting tablet 03cb267b222b440981c94d431e7273bf
0810 18:51:58.398205 (+ 83988us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 18:52:00.668770 (+2270565us) tablet_metadata.cc:641] Metadata flushed
0810 18:52:01.096518 (+427748us) tablet_metadata.cc:641] Metadata flushed
0810 18:52:01.328162 (+231644us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1215255,"tablet-delete.queue_time_us":600304,"thread_start_us":243975,"threads_started":1}
W20260810 18:52:01.375130 19392 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.308s user 0.020s sys 0.015s
I20260810 18:52:01.664040 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:01.669872 16450 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:01.670683 16506 raft_consensus.cc:2396] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260810 18:52:01.671233 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:01.600181 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:01.671932 16750 raft_consensus.cc:3037] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:01.672439 16740 raft_consensus.cc:2396] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260810 18:52:01.611835 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:01.619603 16628 raft_consensus.cc:2379] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 3: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:01.673695 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:01.669876 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:01.389053 16742 consensus_queue.cc:260] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:01.470862 19323 raft_consensus.cc:3037] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:01.674667 16747 raft_consensus.cc:3037] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:01.508718 16495 raft_consensus.cc:2379] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 3: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:01.509847 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:01.680598 17775 raft_consensus.cc:2379] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 3: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
W20260810 18:52:01.682789 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 292423us
I20260810 18:52:01.604179 19473 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:01.596122 16701 leader_election.cc:302] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:01.687902 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:01.677379 19366 raft_consensus.cc:3037] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:01.602923 19554 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:01.785610 16752 raft_consensus.cc:2379] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 5: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:01.789429 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:01.687126 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:01.789664 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 193664us
I20260810 18:52:01.790103 16740 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:01.790887 16628 raft_consensus.cc:2454] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:01.412456 19163 raft_consensus.cc:504] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:01.619683 19475 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:01.687322 16742 raft_consensus.cc:3037] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
W20260810 18:52:01.667335 19392 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.601s user 0.031s sys 0.031s
W20260810 18:52:01.669023 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 301859us
I20260810 18:52:01.799809 16701 leader_election.cc:302] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:01.807227 16750 raft_consensus.cc:2396] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260810 18:52:01.687826 16496 raft_consensus.cc:2362] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3. Re-sending same reply.
W20260810 18:52:01.723851 19533 log.cc:924] Time spent T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.426s user 0.000s sys 0.001s
W20260810 18:52:01.828770 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 185912us
I20260810 18:52:01.837764 16633 raft_consensus.cc:2454] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:01.810575 16746 raft_consensus.cc:2454] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
I20260810 18:52:01.681654 19190 raft_consensus.cc:482] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:01.682889 19495 leader_election.cc:288] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:01.793159 19537 log.cc:924] Time spent T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.448s user 0.000s sys 0.001s
I20260810 18:52:01.818405 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:01.640748 18903 raft_consensus.cc:504] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:01.819151 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:01.791612 19435 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:01.688637 19477 leader_election.cc:288] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:01.798105 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:01.670045 19382 consensus_queue.cc:1035] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.139s
I20260810 18:52:01.687995 19543 raft_consensus.cc:482] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:01.853315 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:01.873927 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:01.828963 19291 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:01.851338 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:01.907648 16746 raft_consensus.cc:2454] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:52:01.908838 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:01.908993 19162 raft_consensus.cc:482] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:01.909648 19162 raft_consensus.cc:504] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:01.910291 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:01.853794 16743 raft_consensus.cc:2454] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:52:01.911824 19099 leader_election.cc:288] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:01.911757 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:01.913429 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:01.913947 16743 raft_consensus.cc:3037] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:01.827863 19129 raft_consensus.cc:2726] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:01.817833 19552 raft_consensus.cc:2726] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:52:01.916767 19391 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:01.816998 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:01.876607 16628 raft_consensus.cc:2396] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260810 18:52:01.844691 16624 consensus_meta.cc:287] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.552s user 0.012s sys 0.000s
I20260810 18:52:01.873783 19435 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:01.919622 16624 raft_consensus.cc:2454] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:01.921039 19435 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:01.881227 19540 log.cc:924] Time spent T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.167s user 0.000s sys 0.000s
I20260810 18:52:01.813957 19475 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:01.923297 19475 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:01.881996 19557 log.cc:924] Time spent T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20260810 18:52:01.882423 19373 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:01.880807 19382 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:01.925444 19382 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:01.926153 16631 raft_consensus.cc:1261] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260810 18:52:01.926988 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:01.882788 19487 log.cc:924] Time spent T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.781s user 0.001s sys 0.001s
I20260810 18:52:01.927549 16631 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:01.885800 16748 raft_consensus.cc:1261] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260810 18:52:01.928188 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:01.928407 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:01.928733 16633 raft_consensus.cc:3037] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:01.929467 16623 raft_consensus.cc:1226] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 LEADER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 2. Current term is 3. Ops: []
I20260810 18:52:01.930303 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:01.931049 16748 raft_consensus.cc:2379] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 3: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
W20260810 18:52:01.931449 19497 log.cc:924] Time spent T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.807s user 0.004s sys 0.000s
I20260810 18:52:01.931896 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:01.932475 16748 raft_consensus.cc:2454] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:52:01.932335 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:01.874656 17787 raft_consensus.cc:2454] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:01.933400 16628 raft_consensus.cc:2379] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 3: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:01.933393 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:01.934283 16748 raft_consensus.cc:2454] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
I20260810 18:52:01.934391 16624 raft_consensus.cc:1226] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 2. Current term is 3. Ops: []
I20260810 18:52:01.889166 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:01.895104 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:01.935858 16752 raft_consensus.cc:2454] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:52:01.936614 16747 raft_consensus.cc:2454] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
I20260810 18:52:01.937341 16752 raft_consensus.cc:1261] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260810 18:52:01.895689 19543 raft_consensus.cc:504] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:01.876999 19477 raft_consensus.cc:2726] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:52:01.899323 19519 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:01.900202 19550 log.cc:924] Time spent T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.282s user 0.001s sys 0.000s
W20260810 18:52:01.909130 19509 log.cc:924] Time spent T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.960s user 0.000s sys 0.002s
I20260810 18:52:01.921169 16750 raft_consensus.cc:2396] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260810 18:52:01.922822 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:01.893852 16636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1353) took 1171 ms. Trace:
W20260810 18:52:01.953040 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 283259us
I20260810 18:52:01.952914 16636 rpcz_store.cc:276] 0810 18:52:00.722857 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:00.723013 (+ 156us) service_pool.cc:227] Handling call
0810 18:52:01.893630 (+1170617us) spinlock_profiling.cc:243] Waited 429 ms on lock 0x7b540035eb54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:01.893773 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":985723648}
I20260810 18:52:01.954137 16450 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:01.851419 19392 raft_consensus.cc:2726] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:01.868088 16740 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:01.947333 16752 raft_consensus.cc:1226] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:02.013545 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:02.014734 16740 raft_consensus.cc:1226] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:02.016142 16747 raft_consensus.cc:1226] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:01.868669 19495 consensus_queue.cc:1046] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.095s
I20260810 18:52:01.869297 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:01.911700 19162 leader_election.cc:288] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:01.803695 19554 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:02.041249 19448 raft_consensus.cc:2726] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:01.947279 19515 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:01.853374 19190 raft_consensus.cc:504] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:01.968290 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:01.975603 16743 raft_consensus.cc:2454] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
I20260810 18:52:02.005302 16742 raft_consensus.cc:1226] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 2. Current term is 3. Ops: []
I20260810 18:52:02.043694 19544 consensus_queue.cc:1035] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260810 18:52:01.944923 16633 raft_consensus.cc:2454] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
W20260810 18:52:02.030202 19556 log.cc:924] Time spent T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.154s user 0.000s sys 0.000s
I20260810 18:52:02.031708 18903 leader_election.cc:288] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:02.032792 19163 leader_election.cc:288] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:02.037205 16447 leader_election.cc:302] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:52:02.039497 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 85434.4us
I20260810 18:52:01.962913 16748 raft_consensus.cc:1226] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260810 18:52:02.047678 16695 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:02.050007 16450 leader_election.cc:302] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:02.041445 19495 raft_consensus.cc:3032] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:52:02.042764 16636 raft_consensus.cc:2454] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:02.008286 19382 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:02.050637 19448 raft_consensus.cc:2726] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:52:02.053980 16502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1344) took 4384 ms. Trace:
I20260810 18:52:02.053819 19495 raft_consensus.cc:729] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:02.056188 19554 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:02.056326 19544 consensus_queue.cc:1035] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260810 18:52:02.056547 19495 consensus_queue.cc:260] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:02.057216 19574 log.cc:924] Time spent T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20260810 18:52:02.052932 16450 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:02.050853 18903 raft_consensus.cc:2726] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:02.058179 19591 raft_consensus.cc:2781] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:02.058683 19591 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:02.059031 19591 raft_consensus.cc:3037] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
W20260810 18:52:02.061647 19534 log.cc:924] Time spent T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.704s user 0.000s sys 0.001s
W20260810 18:52:02.064096 19156 consensus_meta.cc:287] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.704s user 0.003s sys 0.008s
I20260810 18:52:02.064528 19156 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:02.067345 16501 raft_consensus.cc:2354] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 3. Current term is 4.
I20260810 18:52:02.069233 19544 raft_consensus.cc:482] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:02.069727 19544 raft_consensus.cc:504] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:02.072038 19544 leader_election.cc:288] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:02.072259 19351 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:02.072723 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:02.073416 17783 raft_consensus.cc:2454] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
I20260810 18:52:02.073668 19543 leader_election.cc:288] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:02.054355 18728 leader_election.cc:288] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:02.075621 16695 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:02.077044 19190 leader_election.cc:288] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:02.077319 19382 raft_consensus.cc:2781] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:02.077692 19382 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:02.078035 19382 raft_consensus.cc:3037] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:02.053998 19162 raft_consensus.cc:2741] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
W20260810 18:52:02.078663 19504 consensus_meta.cc:287] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.772s user 0.005s sys 0.001s
I20260810 18:52:02.079448 16745 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6fa467f263e640b095b31b6d998258db"
dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
mode: GRACEFUL
new_leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
from {username='slave'} at 127.0.0.1:45870
I20260810 18:52:02.080049 16745 raft_consensus.cc:595] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Received request to transfer leadership to TS 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:02.080355 16745 raft_consensus.cc:603] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260810 18:52:02.055853 16502 rpcz_store.cc:276] 0810 18:51:57.668653 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:57.668805 (+ 152us) service_pool.cc:227] Handling call
0810 18:51:57.669138 (+ 333us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:00.725064 (+3055926us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:02.052397 (+1327333us) spinlock_profiling.cc:243] Waited 3.03 s on lock 0x7b5400145d54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:02.052655 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6973529344}
I20260810 18:52:02.083264 16628 raft_consensus.cc:2396] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260810 18:52:02.083211 16450 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:02.084537 16450 leader_election.cc:302] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:02.085497 16695 leader_election.cc:302] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
W20260810 18:52:02.090652 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 290885us
I20260810 18:52:02.091572 16701 leader_election.cc:302] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:02.092990 16701 leader_election.cc:302] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:02.093012 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 296606us
I20260810 18:52:02.066874 19156 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:02.094484 16450 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
W20260810 18:52:02.094766 17785 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.683s user 0.012s sys 0.006s
I20260810 18:52:02.096418 16701 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:02.049779 19480 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:02.100397 19163 raft_consensus.cc:2726] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:02.101881 16701 leader_election.cc:302] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:02.102807 19163 raft_consensus.cc:2726] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:02.104118 16701 leader_election.cc:302] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:02.105055 19163 raft_consensus.cc:2781] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:52:02.105489 19163 raft_consensus.cc:482] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:02.105902 19163 raft_consensus.cc:3037] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:02.115994 16701 leader_election.cc:302] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:02.117127 19099 raft_consensus.cc:2781] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:52:02.117594 19099 raft_consensus.cc:482] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:02.118031 19099 raft_consensus.cc:3037] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:52:02.119581 16701 leader_election.cc:302] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:02.121042 16701 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:02.129724 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:02.131393 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:52:02.132385 17785 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.720s user 0.012s sys 0.006s
I20260810 18:52:02.089414 19162 raft_consensus.cc:2781] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:02.085254 19350 raft_consensus.cc:2781] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election won for term 3
W20260810 18:52:02.137295 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 744437us
I20260810 18:52:01.357597 17780 rpcz_store.cc:276] 0810 18:51:58.658809 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:58.673488 (+ 14679us) service_pool.cc:227] Handling call
0810 18:51:58.731896 (+ 58408us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:51:58.747100 (+ 15204us) raft_consensus.cc:1520] Early marking committed up to index 6
0810 18:52:00.377847 (+1630747us) log.cc:842] Serialized 14 byte log entry
0810 18:52:01.258991 (+881144us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:01.261635 (+ 2644us) raft_consensus.cc:1631] Marking committed up to 6
0810 18:52:01.262924 (+ 1289us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:01.272679 (+ 9755us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:01.286326 (+ 13647us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1903958,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":35414,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":25475,"thread_start_us":2416599,"threads_started":2,"wal-append.queue_time_us":793330}
I20260810 18:52:02.140816 16632 raft_consensus.cc:1226] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:02.121841 19544 raft_consensus.cc:2781] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260810 18:52:02.133682 17726 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.726s user 0.002s sys 0.012s
W20260810 18:52:02.193055 17726 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.786s user 0.002s sys 0.012s
W20260810 18:52:02.193364 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 804046us
I20260810 18:52:02.194057 17726 leader_election.cc:398] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 election: Vote denied by peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827) with higher term. Message: Invalid argument: T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for earlier term 2. Current term is 3.
I20260810 18:52:02.086618 19504 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:02.122013 19584 raft_consensus.cc:2781] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:52:02.148916 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:02.154938 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:02.168138 16747 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb"
dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
mode: GRACEFUL
new_leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
from {username='slave'} at 127.0.0.1:45870
I20260810 18:52:02.228425 16747 raft_consensus.cc:595] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Received request to transfer leadership to TS 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:02.228770 16747 raft_consensus.cc:603] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260810 18:52:02.170162 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:02.229610 16506 raft_consensus.cc:2454] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 2.
I20260810 18:52:02.155490 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:02.230708 16628 raft_consensus.cc:2454] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:02.151979 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:02.190572 16573 leader_election.cc:302] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:02.191946 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:02.232024 16632 raft_consensus.cc:2379] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 3: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:02.232620 16573 leader_election.cc:302] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:02.144380 17778 raft_consensus.cc:2454] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:02.141770 19480 raft_consensus.cc:2781] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:52:02.122200 18903 raft_consensus.cc:2781] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:52:02.190861 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:02.196807 17726 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:02.237251 16623 raft_consensus.cc:2379] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 3: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:02.238055 17726 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:02.190093 17771 raft_consensus.cc:1261] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260810 18:52:02.238986 19291 consensus_queue.cc:1046] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.009s
I20260810 18:52:02.245244 19291 raft_consensus.cc:3032] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:52:02.245586 19291 raft_consensus.cc:729] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:02.144712 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:02.151885 19162 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:02.247056 19162 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:02.247236 19323 raft_consensus.cc:2781] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:52:02.247767 19323 raft_consensus.cc:686] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:02.248611 19323 consensus_queue.cc:237] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:02.250833 16747 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712"
dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
mode: GRACEFUL
new_leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
from {username='slave'} at 127.0.0.1:45870
I20260810 18:52:02.229094 16631 raft_consensus.cc:3032] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:52:02.251434 16747 raft_consensus.cc:595] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Received request to transfer leadership to TS 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:02.251509 16631 raft_consensus.cc:729] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:02.251827 16747 raft_consensus.cc:603] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260810 18:52:02.252260 16631 consensus_queue.cc:260] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:02.253094 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260810 18:52:02.171783 19495 raft_consensus.cc:3037] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:02.236074 17724 leader_election.cc:302] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:02.239809 17726 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:02.208349 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 87397.6us
I20260810 18:52:02.189446 19279 consensus_queue.cc:1035] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20260810 18:52:02.192961 19544 raft_consensus.cc:482] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:02.235081 19532 consensus_queue.cc:1046] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
I20260810 18:52:02.254813 16750 raft_consensus.cc:1226] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 2. Current term is 3. Ops: []
I20260810 18:52:02.255555 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:52:02.243845 19554 raft_consensus.cc:2781] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260810 18:52:02.283671 19558 log.cc:924] Time spent T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.276s user 0.002s sys 0.000s
I20260810 18:52:02.227253 19584 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:02.266506 16701 leader_election.cc:302] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:02.266927 16573 leader_election.cc:398] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Vote denied by peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791) with higher term. Message: Invalid argument: T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 3. Current term is 4.
I20260810 18:52:02.278175 16447 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:02.267093 17726 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:02.291002 16573 leader_election.cc:401] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260810 18:52:02.281381 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:02.246382 19291 consensus_queue.cc:260] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:02.194077 19350 raft_consensus.cc:686] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:02.240322 19470 raft_consensus.cc:2781] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:02.244318 19578 raft_consensus.cc:3037] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:02.236389 19480 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:02.236624 18903 raft_consensus.cc:482] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:02.331519 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:02.189617 19587 consensus_queue.cc:1035] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.099s
I20260810 18:52:02.291703 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:02.299388 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:02.286015 19544 raft_consensus.cc:3037] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:52:02.303740 17726 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:02.368253 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:02.306867 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:52:02.370460 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 66830.6us
W20260810 18:52:02.256160 19380 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.728s user 0.034s sys 0.035s
I20260810 18:52:02.386319 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:02.316670 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:02.286164 19573 raft_consensus.cc:2781] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:52:02.327858 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:02.330799 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:02.171368 19595 raft_consensus.cc:2781] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:02.327051 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:02.425199 19595 raft_consensus.cc:482] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:02.425877 19595 raft_consensus.cc:3037] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:02.426398 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:02.371076 18903 raft_consensus.cc:3037] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:02.285866 19536 raft_consensus.cc:2726] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:52:02.195441 19163 raft_consensus.cc:504] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:02.286367 19029 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:02.296584 19584 raft_consensus.cc:3037] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:02.255336 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:02.315338 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:52:02.415405 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:02.283077 19471 consensus_queue.cc:1035] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260810 18:52:02.417979 16748 raft_consensus.cc:2396] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 6 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260810 18:52:02.276998 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 436634us
I20260810 18:52:02.390025 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:02.320153 19589 consensus_queue.cc:1046] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.120s
I20260810 18:52:02.316951 19351 raft_consensus.cc:3037] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:02.280292 19190 consensus_queue.cc:1046] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.089s
I20260810 18:52:02.421010 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:02.421360 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:02.320907 18728 raft_consensus.cc:2781] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:02.320750 19577 raft_consensus.cc:2781] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:02.423054 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:02.317175 19519 raft_consensus.cc:3032] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:52:02.423367 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:02.532915 16749 raft_consensus.cc:3037] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:02.267606 19289 consensus_queue.cc:1035] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.124s
I20260810 18:52:02.332288 19480 raft_consensus.cc:3037] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:02.283690 19554 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:02.289149 19524 raft_consensus.cc:2229] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Raft consensus shutting down.
I20260810 18:52:02.429443 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:02.431033 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:02.390704 19380 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.878s user 0.035s sys 0.035s
I20260810 18:52:02.435261 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:02.384194 19470 raft_consensus.cc:686] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:02.417366 19291 raft_consensus.cc:3037] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:02.444741 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:02.551779 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:02.553117 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:02.433229 19366 consensus_meta.cc:287] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.635s user 0.006s sys 0.003s
I20260810 18:52:02.442816 16633 raft_consensus.cc:2454] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 3.
I20260810 18:52:02.418959 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:02.465636 16506 raft_consensus.cc:1799] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Handling vote request from an unknown peer 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:02.594880 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:02.525408 18728 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:02.433089 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:52:02.465639 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 199201us
I20260810 18:52:02.441962 19029 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:02.440474 19606 raft_consensus.cc:3037] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:52:02.415436 19532 consensus_queue.cc:1035] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.040s
I20260810 18:52:02.384047 19392 consensus_queue.cc:1046] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.071s
I20260810 18:52:02.527439 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:02.650825 19366 raft_consensus.cc:3032] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 LEADER]: Stepping down as leader of term 4
I20260810 18:52:02.651368 19366 raft_consensus.cc:729] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:02.651469 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:02.652292 19366 consensus_queue.cc:260] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:02.529512 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 229307us
I20260810 18:52:02.529887 19190 consensus_queue.cc:1225] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Peer d304ac47fdf846de81337057a4ef7bea log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260810 18:52:02.547520 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:02.462044 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:02.391460 19573 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:02.655402 19573 raft_consensus.cc:3037] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:02.527786 16500 raft_consensus.cc:1226] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:02.595415 16624 raft_consensus.cc:2454] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 4.
I20260810 18:52:02.596319 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:02.656365 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:02.656607 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:02.654808 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:02.657768 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:02.531176 19350 consensus_queue.cc:237] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:02.658454 16500 raft_consensus.cc:2362] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5. Re-sending same reply.
I20260810 18:52:02.466235 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:02.658869 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:02.659204 19532 consensus_queue.cc:1225] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer 0397d760589d40d3b92ef141cdfc2dbb log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260810 18:52:02.660328 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:02.597572 17726 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:02.600265 16636 raft_consensus.cc:2396] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260810 18:52:02.660995 16748 raft_consensus.cc:3037] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:02.599635 16506 raft_consensus.cc:2379] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 3: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:02.661577 19494 raft_consensus.cc:2781] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:02.661943 19494 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:02.662274 19494 raft_consensus.cc:3037] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:02.662490 16636 raft_consensus.cc:3032] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 LEADER]: Stepping down as leader of term 4
I20260810 18:52:02.609594 16701 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:02.662928 16636 raft_consensus.cc:729] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:02.600911 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:02.613521 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:02.613942 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:02.542847 19613 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:02.549697 19554 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:02.530790 19577 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:02.599583 18948 consensus_queue.cc:1035] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
W20260810 18:52:02.622704 19626 log.cc:924] Time spent T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
W20260810 18:52:02.623126 19601 log.cc:924] Time spent T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.156s user 0.001s sys 0.000s
I20260810 18:52:02.623502 19504 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:02.624325 19616 log.cc:924] Time spent T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20260810 18:52:02.623907 19470 consensus_queue.cc:237] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:02.545835 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:02.623521 19524 raft_consensus.cc:2258] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Raft consensus is shut down!
I20260810 18:52:02.605228 19543 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:02.629704 19029 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:02.632076 16575 leader_election.cc:334] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Timed out: RequestConsensusVote RPC to 127.14.32.195:40171 timed out after 2.347s (SENT)
W20260810 18:52:02.632315 16621 consensus_meta.cc:287] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.592s user 0.009s sys 0.000s
I20260810 18:52:02.612185 18728 raft_consensus.cc:3037] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:02.639460 19593 raft_consensus.cc:2781] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:02.644269 19628 raft_consensus.cc:2781] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:02.632133 19163 leader_election.cc:288] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:02.652686 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:02.552021 16632 raft_consensus.cc:2362] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3. Re-sending same reply.
I20260810 18:52:02.532634 19519 raft_consensus.cc:729] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
W20260810 18:52:02.682195 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 84727.2us
I20260810 18:52:02.683755 16573 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:02.684633 16636 consensus_queue.cc:260] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:02.710065 19162 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:02.695000 19596 log.cc:924] Time spent T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.264s user 0.002s sys 0.000s
I20260810 18:52:02.620496 19446 consensus_queue.cc:1046] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.091s
I20260810 18:52:02.713064 18808 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:02.691736 19614 log.cc:924] Time spent T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.263s user 0.001s sys 0.001s
I20260810 18:52:02.715140 16750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43726 (request call id 1365) took 1313 ms. Trace:
I20260810 18:52:02.785192 16750 rpcz_store.cc:276] 0810 18:52:01.401973 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:01.957277 (+555304us) service_pool.cc:227] Handling call
0810 18:52:01.960829 (+ 3552us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:02.278230 (+317401us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:02.714889 (+436659us) spinlock_profiling.cc:243] Waited 294 ms on lock 0x7b540034e9d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:02.715059 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":675566464}
I20260810 18:52:02.692400 19613 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:02.716585 19591 consensus_meta.cc:287] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.657s user 0.010s sys 0.003s
I20260810 18:52:02.718217 16450 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:02.836628 16748 raft_consensus.cc:2454] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 5.
W20260810 18:52:02.719316 19382 consensus_meta.cc:287] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.641s user 0.004s sys 0.008s
I20260810 18:52:02.719581 16631 raft_consensus.cc:3037] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:02.694729 19577 raft_consensus.cc:3037] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:02.722967 19473 raft_consensus.cc:3032] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Stepping down as leader of term 4
I20260810 18:52:02.724701 18903 raft_consensus.cc:504] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:02.845186 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 291166us
I20260810 18:52:02.686705 19366 raft_consensus.cc:3037] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
W20260810 18:52:02.701979 19625 log.cc:924] Time spent T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.067s user 0.000s sys 0.004s
I20260810 18:52:02.733035 16505 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56468 (request call id 135) took 1365 ms. Trace:
I20260810 18:52:02.852984 16505 rpcz_store.cc:276] 0810 18:52:01.368073 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:01.376013 (+ 7940us) service_pool.cc:227] Handling call
0810 18:52:01.395964 (+ 19951us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:01.396172 (+ 208us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:01.396195 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:01.599773 (+203578us) log.cc:842] Serialized 26 byte log entry
0810 18:52:01.963645 (+363872us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:01.963698 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:01.963792 (+ 94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:02.711882 (+748090us) spinlock_profiling.cc:243] Waited 28.7 ms on lock 0x7b240018db98. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae502acf4 00007f1ae5028dfb 00007f1ae50b7559 00007f1ae50b71e5 00007f1ae50b6740 00007f1ae7da9578 00007f1ae7da9cd8 00007f1ae7d816e8 00007f1ae7d7caea 00007f1ae7d7bf36
0810 18:52:02.726722 (+ 14840us) raft_consensus.cc:1671] finished
0810 18:52:02.726744 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:02.732969 (+ 6225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":65904384,"thread_start_us":281960,"threads_started":1,"wal-append.queue_time_us":440513}
I20260810 18:52:02.750380 16695 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:02.707636 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:02.768257 16636 raft_consensus.cc:3037] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
W20260810 18:52:02.771173 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 87507.8us
I20260810 18:52:02.773484 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:02.752715 19543 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:02.774641 19392 raft_consensus.cc:2726] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:02.774796 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:02.763506 19595 raft_consensus.cc:504] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:02.776885 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 144837us
I20260810 18:52:02.844978 19473 raft_consensus.cc:729] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:02.776453 19519 consensus_queue.cc:260] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:02.772325 16749 raft_consensus.cc:2454] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:02.755415 16621 raft_consensus.cc:2454] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:02.762988 19593 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:02.763998 19628 raft_consensus.cc:482] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:02.841641 17785 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55570 (request call id 381) took 1507 ms. Trace:
W20260810 18:52:03.128118 19633 log.cc:924] Time spent T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.296s user 0.003s sys 0.004s
I20260810 18:52:03.128074 17785 rpcz_store.cc:276] 0810 18:52:01.334110 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:01.334300 (+ 190us) service_pool.cc:227] Handling call
0810 18:52:01.354893 (+ 20593us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:01.355140 (+ 247us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:01.355176 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:01.369513 (+ 14337us) log.cc:842] Serialized 26 byte log entry
0810 18:52:02.145221 (+775708us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:02.145274 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:02.145364 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:02.841399 (+696035us) raft_consensus.cc:1671] finished
0810 18:52:02.841423 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:02.841573 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":732938,"threads_started":1,"wal-append.queue_time_us":1002143}
W20260810 18:52:02.781509 19495 consensus_meta.cc:287] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.512s user 0.010s sys 0.004s
I20260810 18:52:02.730629 16626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1409) took 1477 ms. Trace:
I20260810 18:52:03.129918 16743 raft_consensus.cc:2454] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:02.441540 16323 catalog_manager.cc:4841] TS d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): tablet 03cb267b222b440981c94d431e7273bf (table test-workload [id=5aa4952cbee8447ca410249e65af5af6]) successfully deleted
I20260810 18:52:03.129716 16626 rpcz_store.cc:276] 0810 18:52:01.253045 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:02.032967 (+779922us) service_pool.cc:227] Handling call
0810 18:52:02.033421 (+ 454us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:02.033896 (+ 475us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:02.033939 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:02.034486 (+ 547us) log.cc:842] Serialized 26 byte log entry
0810 18:52:02.372908 (+338422us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:52:02.373013 (+ 105us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:02.373103 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:02.730419 (+357316us) raft_consensus.cc:1671] finished
0810 18:52:02.730440 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:02.730573 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":121,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":110,"thread_start_us":248919,"threads_started":1,"wal-append.queue_time_us":403322}
I20260810 18:52:03.130089 19628 raft_consensus.cc:3037] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
W20260810 18:52:02.855458 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 105191us
W20260810 18:52:02.836452 19099 consensus_meta.cc:287] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.718s user 0.009s sys 0.003s
I20260810 18:52:02.834334 19578 raft_consensus.cc:2726] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:03.135159 19099 raft_consensus.cc:504] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:52:03.135622 17787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55586 (request call id 281) took 1065 ms. Trace:
I20260810 18:52:03.136265 16747 raft_consensus.cc:1226] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 1. Current term is 2. Ops: []
W20260810 18:52:03.136586 19584 consensus_meta.cc:287] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.678s user 0.012s sys 0.005s
I20260810 18:52:03.136636 19495 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:02.891831 17726 leader_election.cc:302] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:03.137276 19099 leader_election.cc:288] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:02.873167 16636 raft_consensus.cc:1261] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20260810 18:52:03.138130 16636 pending_rounds.cc:76] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb: Aborting all ops after (but not including) 3
W20260810 18:52:03.138027 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 246297us
I20260810 18:52:03.135983 17787 rpcz_store.cc:276] 0810 18:52:02.070053 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:02.070685 (+ 632us) service_pool.cc:227] Handling call
0810 18:52:02.071133 (+ 448us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:02.071510 (+ 377us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:02.071541 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:02.071973 (+ 432us) log.cc:842] Serialized 26 byte log entry
0810 18:52:02.285094 (+213121us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:02.285129 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:02.285196 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:03.135433 (+850237us) raft_consensus.cc:1671] finished
0810 18:52:03.135453 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:03.135568 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":13246,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":9050,"thread_start_us":212815,"threads_started":1,"wal-append.queue_time_us":552702}
I20260810 18:52:02.917588 16749 raft_consensus.cc:1226] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 2. Current term is 3. Ops: []
I20260810 18:52:03.139339 19114 raft_consensus.cc:2781] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:03.139870 19114 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:03.140328 19114 raft_consensus.cc:3037] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:03.140424 16749 pending_rounds.cc:76] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be: Aborting all ops after (but not including) 1
I20260810 18:52:03.140877 16749 pending_rounds.cc:97] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260810 18:52:03.141135 16749 raft_consensus.cc:2866] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:52:02.864509 19635 raft_consensus.cc:2781] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:03.142900 19635 raft_consensus.cc:686] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:03.143940 19635 consensus_queue.cc:237] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:02.844313 19591 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:03.149082 19591 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:02.919131 17786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55586 (request call id 277) took 1575 ms. Trace:
I20260810 18:52:03.137326 19495 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:03.149729 17786 rpcz_store.cc:276] 0810 18:52:01.345870 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:01.346027 (+ 157us) service_pool.cc:227] Handling call
0810 18:52:01.346334 (+ 307us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:01.681677 (+335343us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:01.681713 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:01.682142 (+ 429us) log.cc:842] Serialized 26 byte log entry
0810 18:52:02.085715 (+403573us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:02.085748 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:02.085822 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:02.899161 (+813339us) raft_consensus.cc:1671] finished
0810 18:52:02.899186 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:02.919038 (+ 19852us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12260,"raft.queue_time_us":28314,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":6533,"thread_start_us":604327,"threads_started":2,"wal-append.queue_time_us":500451}
W20260810 18:52:02.942505 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 48925.8us
I20260810 18:52:03.151584 19495 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:02.871512 19279 raft_consensus.cc:2726] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:02.840444 19382 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:03.152961 16740 raft_consensus.cc:2379] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 4: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:02.863427 17780 raft_consensus.cc:2454] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
I20260810 18:52:02.883349 19190 raft_consensus.cc:482] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:02.893851 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 284345us
I20260810 18:52:03.154631 16701 leader_election.cc:302] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:02.915597 19504 raft_consensus.cc:2741] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260810 18:52:02.921090 19593 raft_consensus.cc:3037] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:03.168424 19382 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:03.169059 19382 consensus_queue.cc:1035] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:52:03.171947 19392 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:03.172559 19392 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:03.182914 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:03.050972 16495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56468 (request call id 137) took 1014 ms. Trace:
W20260810 18:52:03.115113 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 247465us
I20260810 18:52:02.919696 16633 raft_consensus.cc:2396] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260810 18:52:03.117259 19519 raft_consensus.cc:3037] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:02.917178 16631 raft_consensus.cc:1261] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:52:03.118196 17781 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55570 (request call id 376) took 2951 ms. Trace:
I20260810 18:52:03.119344 16752 raft_consensus.cc:1226] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: []
W20260810 18:52:03.119563 19351 consensus_meta.cc:287] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.653s user 0.011s sys 0.000s
I20260810 18:52:03.120285 19613 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:03.207178 19392 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:03.120373 19544 consensus_meta.cc:287] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.698s user 0.009s sys 0.005s
I20260810 18:52:03.221565 19392 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:03.217018 19584 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:03.120957 19637 log.cc:924] Time spent T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.163s user 0.003s sys 0.003s
I20260810 18:52:03.222262 19392 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:03.223626 19584 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:03.121923 16621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 1421) took 1333 ms. Trace:
I20260810 18:52:02.950654 17727 leader_election.cc:302] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:03.123409 19494 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:02.921077 19168 consensus_queue.cc:1035] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.105s
W20260810 18:52:03.124677 19480 consensus_meta.cc:287] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.569s user 0.009s sys 0.004s
I20260810 18:52:03.126973 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:03.127386 19524 ts_tablet_manager.cc:1905] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 18:52:02.895581 18948 consensus_queue.cc:1035] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.083s
W20260810 18:52:02.846024 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 127872us
I20260810 18:52:03.128999 19473 consensus_queue.cc:260] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:03.132035 16323 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 837789us
I20260810 18:52:03.138430 16636 pending_rounds.cc:97] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260810 18:52:03.248441 16636 raft_consensus.cc:2866] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:52:03.157953 16701 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
W20260810 18:52:03.006117 19632 log.cc:924] Time spent T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.157s user 0.000s sys 0.000s
I20260810 18:52:03.159308 18808 raft_consensus.cc:2726] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:03.160012 19190 raft_consensus.cc:504] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:03.117745 19573 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:03.213469 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:03.221776 19544 raft_consensus.cc:504] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:03.275516 17785 raft_consensus.cc:3037] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:03.276731 19544 leader_election.cc:288] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:03.276780 19573 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:03.277459 16745 raft_consensus.cc:1226] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 1. Current term is 2. Ops: []
I20260810 18:52:03.242343 19480 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:03.279322 16496 raft_consensus.cc:2454] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 3.
I20260810 18:52:03.281212 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:03.282140 16508 raft_consensus.cc:2379] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 3: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:03.284514 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:03.285666 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:03.287247 19480 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:03.287741 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:03.288657 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:03.289335 16508 raft_consensus.cc:3037] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:03.286748 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:03.207398 16495 rpcz_store.cc:276] 0810 18:52:02.036655 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:02.038773 (+ 2118us) service_pool.cc:227] Handling call
0810 18:52:02.045906 (+ 7133us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:02.046150 (+ 244us) raft_consensus.cc:1520] Early marking committed up to index 5
0810 18:52:02.418374 (+372224us) log.cc:842] Serialized 14 byte log entry
0810 18:52:02.895119 (+476745us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:02.895208 (+ 89us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:02.897888 (+ 2680us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:02.918713 (+ 20825us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:03.000131 (+ 81418us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":479919,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":44,"raft.queue_time_us":33641,"raft.run_cpu_time_us":74,"raft.run_wall_time_us":74,"thread_start_us":717160,"threads_started":3,"wal-append.queue_time_us":339222}
I20260810 18:52:03.120926 19595 leader_election.cc:288] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:03.221307 19351 raft_consensus.cc:2726] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791) with higher term. Message: Invalid argument: T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 3. Current term is 4.
I20260810 18:52:03.232311 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:03.233265 18903 leader_election.cc:288] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:03.216650 17781 rpcz_store.cc:276] 0810 18:52:00.167020 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:00.169413 (+ 2393us) service_pool.cc:227] Handling call
0810 18:52:01.158685 (+989272us) spinlock_profiling.cc:243] Waited 86.3 ms on lock 0x7b5800005498. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed0502b0 00007f1aed03b7f9 00007f1aed03b2a2 00007f1ae6721684
0810 18:52:01.188513 (+ 29828us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:01.463701 (+275188us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:01.470920 (+ 7219us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:01.688256 (+217336us) log.cc:842] Serialized 26 byte log entry
0810 18:52:02.327576 (+639320us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:02.367275 (+ 39699us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:03.117842 (+750567us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x7b54006b1554. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:03.117907 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:03.117972 (+ 65us) raft_consensus.cc:1671] finished
0810 18:52:03.117984 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:03.118132 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12799,"raft.queue_time_us":454785,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":74457,"spinlock_wait_cycles":659277952,"thread_start_us":454068,"threads_started":1,"wal-append.queue_time_us":639601}
I20260810 18:52:03.233703 16448 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:03.231235 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:03.242372 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:03.230849 19644 log.cc:924] Time spent T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.093s user 0.000s sys 0.001s
I20260810 18:52:03.231122 19392 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:03.245288 19380 raft_consensus.cc:2781] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:03.230310 16621 rpcz_store.cc:276] 0810 18:52:01.788830 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:01.793027 (+ 4197us) service_pool.cc:227] Handling call
0810 18:52:02.920534 (+1127507us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:03.257128 19162 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:03.256062 19099 raft_consensus.cc:2932] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:52:03.258399 16742 raft_consensus.cc:1226] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 2. Current term is 3. Ops: []
I20260810 18:52:03.258800 19495 raft_consensus.cc:2726] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260810 18:52:03.260578 19190 leader_election.cc:288] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:03.261214 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 103375us
I20260810 18:52:03.341071 16701 leader_election.cc:302] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:03.262009 19473 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:03.262133 17783 raft_consensus.cc:2379] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 4: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:52:03.342594 16701 leader_election.cc:302] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:03.344873 16701 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:03.267338 19543 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:03.267870 16695 leader_election.cc:302] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:03.268640 19244 consensus_queue.cc:1046] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
W20260810 18:52:03.271641 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 355503us
I20260810 18:52:03.332214 19350 raft_consensus.cc:2781] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:03.296420 17785 raft_consensus.cc:2454] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:03.306298 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:03.311458 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:03.312999 17787 raft_consensus.cc:2454] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:52:03.313016 16501 raft_consensus.cc:2454] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:03.320482 16628 raft_consensus.cc:2454] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
W20260810 18:52:03.327519 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 94019us
I20260810 18:52:03.331861 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:03.365181 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:03.365942 17781 raft_consensus.cc:2354] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4.
I20260810 18:52:03.264931 19114 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:03.368940 19114 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:03.330698 19640 log.cc:924] Time spent T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.189s user 0.000s sys 0.001s
I20260810 18:52:03.370281 17786 raft_consensus.cc:1226] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 4. Current term is 5. Ops: []
W20260810 18:52:03.347785 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 79983.2us
I20260810 18:52:03.372186 16695 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:03.349835 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:03.374095 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:03.374539 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:03.375306 19552 raft_consensus.cc:2781] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:52:03.375813 19552 raft_consensus.cc:686] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:03.376636 19552 consensus_queue.cc:237] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:03.381850 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:03.382987 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:03.383878 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:03.385365 16502 raft_consensus.cc:1226] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:03.386991 17780 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55602 (request call id 308) took 1076 ms. Trace:
I20260810 18:52:03.335451 19380 raft_consensus.cc:686] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 5 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:03.387401 17780 rpcz_store.cc:276] 0810 18:52:02.310222 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:02.315733 (+ 5511us) service_pool.cc:227] Handling call
0810 18:52:03.386836 (+1071103us) spinlock_profiling.cc:243] Waited 237 ms on lock 0x7b540069da54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:03.386935 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":545599872}
I20260810 18:52:03.388396 19380 consensus_queue.cc:237] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:03.389868 16746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1483) took 1097 ms. Trace:
I20260810 18:52:03.390226 16746 rpcz_store.cc:276] 0810 18:52:02.292422 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:02.544309 (+251887us) service_pool.cc:227] Handling call
0810 18:52:02.605747 (+ 61438us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:02.605970 (+ 223us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:02.605994 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:02.606487 (+ 493us) log.cc:842] Serialized 26 byte log entry
0810 18:52:02.862818 (+256331us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:52:02.862869 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:02.862958 (+ 89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:03.389659 (+526701us) raft_consensus.cc:1671] finished
0810 18:52:03.389677 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:03.389815 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":219928,"threads_started":1,"wal-append.queue_time_us":499245}
I20260810 18:52:03.394131 16447 leader_election.cc:398] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Vote denied by peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387) with higher term. Message: Invalid argument: T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4.
I20260810 18:52:03.394446 16447 leader_election.cc:401] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260810 18:52:03.361788 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:03.351110 19495 raft_consensus.cc:2726] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:52:03.325506 16508 raft_consensus.cc:2454] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 3.
I20260810 18:52:03.401433 16508 raft_consensus.cc:1226] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3. Ops: []
I20260810 18:52:03.296155 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:03.346678 19646 raft_consensus.cc:2726] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260810 18:52:03.409009 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 125429us
I20260810 18:52:03.350524 19595 raft_consensus.cc:2726] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:52:03.346757 18903 raft_consensus.cc:2726] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260810 18:52:03.400442 19606 consensus_meta.cc:287] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.753s user 0.005s sys 0.010s
I20260810 18:52:03.399282 16742 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:03.348052 16701 leader_election.cc:302] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:03.362982 19350 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
W20260810 18:52:03.411761 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 48478.3us
I20260810 18:52:03.412740 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:03.407473 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:03.424695 16501 raft_consensus.cc:2454] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
W20260810 18:52:03.421967 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 74012.9us
I20260810 18:52:03.426082 16701 leader_election.cc:302] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:03.419050 17724 leader_election.cc:302] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:03.417054 16573 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:52:03.428617 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:03.429464 16636 raft_consensus.cc:2454] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 3.
I20260810 18:52:03.434722 16573 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters: d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:03.414031 19554 consensus_meta.cc:287] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.706s user 0.006s sys 0.007s
I20260810 18:52:03.444985 16448 leader_election.cc:302] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:03.437119 16573 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:03.438341 16495 raft_consensus.cc:1261] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260810 18:52:03.447057 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:03.483292 16636 raft_consensus.cc:3037] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:03.431648 19473 raft_consensus.cc:3037] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:03.433764 19606 raft_consensus.cc:2726] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:52:03.513141 17778 raft_consensus.cc:3037] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:03.513478 17726 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:03.431303 19350 raft_consensus.cc:3037] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
W20260810 18:52:03.517232 18728 consensus_meta.cc:287] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.748s user 0.008s sys 0.007s
I20260810 18:52:03.517655 18728 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:03.519824 18728 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:03.424531 19323 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.881s user 0.020s sys 0.029s
W20260810 18:52:03.523159 19323 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.988s user 0.021s sys 0.030s
I20260810 18:52:03.523732 19323 raft_consensus.cc:2781] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:03.524080 19323 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:03.524401 19323 raft_consensus.cc:3037] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:03.524480 19573 raft_consensus.cc:482] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:03.525010 19573 raft_consensus.cc:504] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:03.527405 19573 leader_election.cc:288] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:03.535854 19350 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:03.507463 19392 raft_consensus.cc:2781] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:03.434341 19351 raft_consensus.cc:482] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:03.503453 19446 raft_consensus.cc:2726] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:03.435137 18903 raft_consensus.cc:2781] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:03.538343 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:03.539103 19595 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:03.539564 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:03.540979 16632 raft_consensus.cc:1261] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260810 18:52:03.584281 16632 pending_rounds.cc:76] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb: Aborting all ops after (but not including) 1
I20260810 18:52:03.584604 16632 pending_rounds.cc:97] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260810 18:52:03.584860 16632 raft_consensus.cc:2866] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:52:03.541160 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:03.586159 16623 raft_consensus.cc:2454] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:03.541574 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:03.543417 16742 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
W20260810 18:52:03.545217 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 108177us
I20260810 18:52:03.458461 18808 raft_consensus.cc:2781] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:03.544785 19635 consensus_queue.cc:1035] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.074s
I20260810 18:52:03.555123 16636 raft_consensus.cc:2454] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 5.
I20260810 18:52:03.458861 19114 raft_consensus.cc:2781] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader election won for term 3
W20260810 18:52:03.561256 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 101732us
I20260810 18:52:03.488353 19494 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:03.557219 19577 consensus_meta.cc:287] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.703s user 0.010s sys 0.001s
I20260810 18:52:03.560025 19543 raft_consensus.cc:2781] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:03.582598 16626 raft_consensus.cc:2379] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 3: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:03.584344 16631 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:03.458070 19554 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:03.568282 16621 raft_consensus.cc:1226] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260810 18:52:03.588858 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:03.585119 18903 raft_consensus.cc:482] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:03.589375 16573 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:03.605696 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:03.606607 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:03.632020 16636 raft_consensus.cc:2454] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 3.
I20260810 18:52:03.607900 16448 leader_election.cc:302] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:03.575470 19392 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:03.612267 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:52:03.587774 19366 consensus_meta.cc:287] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.734s user 0.010s sys 0.000s
I20260810 18:52:03.581650 19679 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:03.632097 16632 raft_consensus.cc:3037] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:03.639810 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:03.589781 16575 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:03.567123 19628 raft_consensus.cc:504] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:03.590502 18808 raft_consensus.cc:482] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:03.647265 16622 raft_consensus.cc:2354] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 4. Current term is 5.
I20260810 18:52:03.639437 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:03.647974 17785 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:03.640470 16631 raft_consensus.cc:2454] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:03.642292 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:03.636485 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:03.651974 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:03.652799 16742 raft_consensus.cc:2396] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260810 18:52:03.587450 19351 raft_consensus.cc:504] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:03.623627 19543 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:03.660454 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:03.617571 19114 raft_consensus.cc:686] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 3 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:03.658867 16636 raft_consensus.cc:2396] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260810 18:52:03.648411 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 101915us
I20260810 18:52:03.663738 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:03.664783 17785 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
W20260810 18:52:03.665320 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 57531us
I20260810 18:52:03.674090 16448 leader_election.cc:302] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:03.665664 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:03.657914 18808 raft_consensus.cc:3037] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:03.601603 19595 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:03.642376 19673 raft_consensus.cc:2726] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:03.637939 18903 raft_consensus.cc:3037] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:03.639269 19392 raft_consensus.cc:3037] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
W20260810 18:52:03.675845 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 86156.7us
I20260810 18:52:03.678164 16632 raft_consensus.cc:2454] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 4.
I20260810 18:52:03.681073 16623 raft_consensus.cc:2454] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
I20260810 18:52:03.669945 16742 raft_consensus.cc:2362] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3. Re-sending same reply.
I20260810 18:52:03.668897 19366 raft_consensus.cc:2726] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:03.669312 19684 raft_consensus.cc:2726] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:03.687498 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:03.622244 19577 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:03.691156 16742 raft_consensus.cc:2379] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 3: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:03.671759 19543 raft_consensus.cc:3037] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
W20260810 18:52:03.695609 19696 log.cc:924] Time spent T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260810 18:52:03.695933 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:03.696566 17785 raft_consensus.cc:3037] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:03.697760 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:03.698555 16749 raft_consensus.cc:2379] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 5: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:03.699014 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:03.699365 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:03.699597 16633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 1457) took 1154 ms. Trace:
I20260810 18:52:03.700189 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:03.701218 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:03.701844 17786 raft_consensus.cc:1226] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:03.702414 19350 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:03.703280 16624 pending_rounds.cc:76] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb: Aborting all ops after (but not including) 4
I20260810 18:52:03.703634 16624 pending_rounds.cc:97] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb: Aborting uncommitted NO_OP operation due to leader change: 3.5
W20260810 18:52:03.703815 19677 log.cc:924] Time spent T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.202s user 0.001s sys 0.000s
I20260810 18:52:03.703918 16624 raft_consensus.cc:2866] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:52:03.646221 19679 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:03.706424 17785 raft_consensus.cc:2454] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:03.706601 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:03.707631 19552 consensus_queue.cc:1046] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 18:52:03.708503 19552 raft_consensus.cc:3032] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 LEADER]: Stepping down as leader of term 4
I20260810 18:52:03.708461 19635 consensus_queue.cc:1046] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:52:03.708638 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:03.709024 19552 raft_consensus.cc:729] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:03.709530 16742 raft_consensus.cc:2379] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 5: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:03.709754 19673 consensus_queue.cc:1035] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:52:03.709864 19552 consensus_queue.cc:260] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:03.710347 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:03.688206 16575 leader_election.cc:302] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:03.711100 19552 raft_consensus.cc:3037] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:03.668661 19660 consensus_queue.cc:1046] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.065s
I20260810 18:52:03.711995 16575 leader_election.cc:302] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:03.689152 16448 leader_election.cc:302] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:03.700009 16749 raft_consensus.cc:2362] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5. Re-sending same reply.
I20260810 18:52:03.701853 16746 raft_consensus.cc:2379] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 5: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:03.714862 16742 raft_consensus.cc:2354] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 3. Current term is 4.
I20260810 18:52:03.715720 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:03.716908 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:03.717813 19708 raft_consensus.cc:482] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:03.718053 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:03.718516 19708 raft_consensus.cc:504] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:03.719451 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:03.720599 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:03.721127 19708 leader_election.cc:288] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:03.721752 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:03.721856 19708 raft_consensus.cc:482] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:03.722878 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:03.694317 19657 raft_consensus.cc:3032] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:52:03.727396 19657 raft_consensus.cc:729] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:03.728369 19657 consensus_queue.cc:260] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:03.729301 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:03.729974 19657 raft_consensus.cc:3037] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:03.730242 17771 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55602 (request call id 312) took 1334 ms. Trace:
I20260810 18:52:03.731896 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:03.730852 16743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1497) took 1013 ms. Trace:
I20260810 18:52:03.733201 16743 rpcz_store.cc:276] 0810 18:52:02.717730 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:03.510139 (+792409us) service_pool.cc:227] Handling call
0810 18:52:03.510533 (+ 394us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:03.512023 (+ 1490us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:03.512071 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:03.512672 (+ 601us) log.cc:842] Serialized 26 byte log entry
0810 18:52:03.663002 (+150330us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:52:03.663048 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:03.663139 (+ 91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:03.730603 (+ 67464us) raft_consensus.cc:1671] finished
0810 18:52:03.730626 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:03.730785 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":804,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":142,"thread_start_us":149907,"threads_started":1,"wal-append.queue_time_us":215783}
I20260810 18:52:03.734992 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:03.736334 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:03.736874 16743 raft_consensus.cc:2454] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 1.
I20260810 18:52:03.737630 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:03.738135 16743 raft_consensus.cc:2362] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3. Re-sending same reply.
I20260810 18:52:03.742008 19351 leader_election.cc:288] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:03.699980 16633 rpcz_store.cc:276] 0810 18:52:02.544939 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:02.596039 (+ 51100us) service_pool.cc:227] Handling call
0810 18:52:03.699393 (+1103354us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b54002ff8d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:03.699538 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":553846016}
I20260810 18:52:03.743676 19628 leader_election.cc:288] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:03.744313 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:03.745541 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:03.745666 19554 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:03.746658 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:03.726760 19366 raft_consensus.cc:2726] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:52:03.746999 19244 raft_consensus.cc:2726] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:52:03.738894 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:03.759124 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:03.694146 19595 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:03.762549 16743 raft_consensus.cc:1226] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 2. Current term is 3. Ops: []
W20260810 18:52:03.751466 19601 log.cc:924] Time spent T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20260810 18:52:03.706964 19679 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:03.766665 16628 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 1452) took 1415 ms (client timeout 1523 ms). Trace:
I20260810 18:52:03.665381 19593 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:03.768194 16743 raft_consensus.cc:1261] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:52:03.778527 16447 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:03.769562 16742 raft_consensus.cc:1261] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260810 18:52:03.778898 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 89785.2us
I20260810 18:52:03.771649 16749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 150) took 1042 ms. Trace:
I20260810 18:52:03.722560 19708 raft_consensus.cc:504] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:03.780411 16448 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:03.780292 16749 rpcz_store.cc:276] 0810 18:52:02.729322 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:03.760591 (+1031269us) service_pool.cc:227] Handling call
0810 18:52:03.771188 (+ 10597us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:03.771356 (+ 168us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:03.771378 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:03.771414 (+ 36us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:03.771439 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:03.771487 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:03.771596 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 18:52:03.778100 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 136770us
I20260810 18:52:03.782579 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:03.773176 16626 raft_consensus.cc:2454] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
W20260810 18:52:03.777768 16628 rpcz_store.cc:269] 0810 18:52:02.367517 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:02.367957 (+ 440us) service_pool.cc:227] Handling call
0810 18:52:03.766467 (+1398510us) spinlock_profiling.cc:243] Waited 937 ms on lock 0x7b5400173754. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:03.766594 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2154805120}
I20260810 18:52:03.783741 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:03.783411 16749 raft_consensus.cc:2354] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for earlier term 4. Current term is 5.
W20260810 18:52:03.788167 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 97925.4us
I20260810 18:52:03.762907 19366 raft_consensus.cc:2726] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:52:03.790623 16740 raft_consensus.cc:1226] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:03.774456 19448 raft_consensus.cc:3032] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:52:03.782153 17780 raft_consensus.cc:1226] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:03.795276 19448 raft_consensus.cc:729] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:03.794540 19471 raft_consensus.cc:2781] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:03.794466 19573 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:03.794056 16505 raft_consensus.cc:1261] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260810 18:52:03.796142 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:03.802323 16505 pending_rounds.cc:76] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea: Aborting all ops after (but not including) 1
I20260810 18:52:03.802641 16505 pending_rounds.cc:97] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260810 18:52:03.797505 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:03.798332 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:03.798249 19448 consensus_queue.cc:260] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:03.799150 19694 log.cc:924] Time spent T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.087s user 0.000s sys 0.001s
I20260810 18:52:03.771215 17778 raft_consensus.cc:2454] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:03.799253 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:03.799474 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:03.808996 16501 raft_consensus.cc:2354] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for earlier term 4. Current term is 5.
I20260810 18:52:03.785995 19684 raft_consensus.cc:2726] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:52:03.809764 16450 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:03.810668 16450 leader_election.cc:302] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:03.769038 19029 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:03.730695 17771 rpcz_store.cc:276] 0810 18:52:02.395410 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:02.441657 (+ 46247us) service_pool.cc:227] Handling call
0810 18:52:03.729077 (+1287420us) spinlock_profiling.cc:243] Waited 5.11 ms on lock 0x7b5800005498. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed03c1b0 00007f1aed03ba94 00007f1ae6723c04 00007f1ae6723b92
0810 18:52:03.730178 (+ 1101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11762944}
I20260810 18:52:03.802902 16505 raft_consensus.cc:2866] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:52:03.813849 17724 leader_election.cc:398] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: Vote denied by peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791) with higher term. Message: Invalid argument: T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for earlier term 4. Current term is 5.
I20260810 18:52:03.815913 17724 leader_election.cc:401] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260810 18:52:03.688220 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:03.819716 16750 raft_consensus.cc:2396] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260810 18:52:03.819897 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:03.820433 16573 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:03.828195 16448 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
W20260810 18:52:03.831488 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 119607us
I20260810 18:52:03.836944 16575 leader_election.cc:302] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:03.838155 19721 log.cc:924] Time spent T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.110s user 0.001s sys 0.000s
I20260810 18:52:03.826493 19606 raft_consensus.cc:2781] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:03.839577 19279 consensus_queue.cc:1035] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:52:03.839808 19543 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:03.841310 16496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56468 (request call id 138) took 1699 ms. Trace:
I20260810 18:52:03.841933 19543 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:03.841722 16496 rpcz_store.cc:276] 0810 18:52:02.142379 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:02.144817 (+ 2438us) service_pool.cc:227] Handling call
0810 18:52:03.841145 (+1696328us) spinlock_profiling.cc:243] Waited 833 ms on lock 0x7b54000bd954. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:03.841251 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1916603264}
I20260810 18:52:03.835172 19029 raft_consensus.cc:2726] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791) with higher term. Message: Invalid argument: T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for earlier term 4. Current term is 5.
I20260810 18:52:03.808508 19573 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:03.843930 17724 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:03.845302 19029 raft_consensus.cc:2741] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260810 18:52:03.846336 19708 leader_election.cc:288] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:03.817178 19373 raft_consensus.cc:2726] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:03.806304 19471 raft_consensus.cc:686] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:03.851567 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:03.852376 16496 raft_consensus.cc:3037] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:03.852566 16575 leader_election.cc:302] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:03.852927 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:03.853474 19543 raft_consensus.cc:2781] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:52:03.853992 19543 raft_consensus.cc:482] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:03.854395 19543 raft_consensus.cc:3037] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:03.822686 19168 raft_consensus.cc:2781] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:52:03.817377 17781 raft_consensus.cc:1226] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:03.860486 16575 leader_election.cc:302] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:03.862565 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:03.863750 16496 raft_consensus.cc:2454] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
I20260810 18:52:03.859054 16450 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:03.826895 19496 raft_consensus.cc:2781] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:52:03.864969 16573 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:03.869777 19708 raft_consensus.cc:2781] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:03.851950 19446 consensus_queue.cc:1035] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20260810 18:52:03.789175 19114 consensus_queue.cc:237] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:03.837073 19678 raft_consensus.cc:2726] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:52:03.874529 16746 raft_consensus.cc:2396] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20260810 18:52:03.889294 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 61170.9us
I20260810 18:52:03.911253 16626 raft_consensus.cc:1226] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 1. Current term is 2. Ops: []
I20260810 18:52:03.856647 19473 raft_consensus.cc:2726] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387) with higher term. Message: Invalid argument: T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4.
I20260810 18:52:03.878664 19587 raft_consensus.cc:2726] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:03.875691 19168 raft_consensus.cc:686] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:03.918785 19577 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:03.870167 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:03.919327 19168 consensus_queue.cc:237] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:03.881400 19593 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:03.860630 19448 raft_consensus.cc:3037] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:52:03.836663 19366 consensus_queue.cc:1035] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.078s
I20260810 18:52:03.922739 16575 leader_election.cc:302] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:03.879771 18948 raft_consensus.cc:2781] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:03.925839 19678 raft_consensus.cc:2726] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260810 18:52:03.878456 19099 consensus_meta.cc:287] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.531s user 0.006s sys 0.002s
I20260810 18:52:03.826872 18728 consensus_queue.cc:1046] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.094s
I20260810 18:52:03.933373 16575 leader_election.cc:302] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:03.879221 19713 raft_consensus.cc:2726] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260810 18:52:03.924017 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 59348us
I20260810 18:52:03.869560 19496 raft_consensus.cc:482] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:03.870303 19708 raft_consensus.cc:482] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:03.942581 16575 leader_election.cc:302] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:03.946960 17787 raft_consensus.cc:1226] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 3 LEADER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3. Ops: []
I20260810 18:52:03.952222 16747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1473) took 1664 ms. Trace:
I20260810 18:52:03.948763 16573 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:03.931368 19715 log.cc:924] Time spent T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260810 18:52:03.952658 16747 rpcz_store.cc:276] 0810 18:52:02.287412 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:02.289012 (+ 1600us) service_pool.cc:227] Handling call
0810 18:52:02.294862 (+ 5850us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:03.136667 (+841805us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:03.951924 (+815257us) spinlock_profiling.cc:243] Waited 841 ms on lock 0x7b540007e9d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:03.952149 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1934557568}
I20260810 18:52:03.881767 19323 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:03.880657 18808 raft_consensus.cc:504] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:03.953400 19496 raft_consensus.cc:3037] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:03.940862 17727 leader_election.cc:398] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: Vote denied by peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171) with higher term. Message: Invalid argument: T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for earlier term 4. Current term is 5.
I20260810 18:52:03.960434 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:52:03.975781 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 96229.4us
I20260810 18:52:03.962175 16636 raft_consensus.cc:1226] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260810 18:52:03.977129 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:03.977681 16636 raft_consensus.cc:3037] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:03.934257 19723 consensus_queue.cc:1046] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260810 18:52:03.966053 19373 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:03.966650 19718 raft_consensus.cc:2781] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:03.955428 19708 raft_consensus.cc:3037] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:03.939409 19660 raft_consensus.cc:2726] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:52:03.939941 19573 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:03.939019 19593 consensus_queue.cc:1046] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.038s
I20260810 18:52:03.875658 19471 consensus_queue.cc:237] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:03.955933 18948 raft_consensus.cc:686] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:03.950906 19679 raft_consensus.cc:2726] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260810 18:52:04.006598 16628 raft_consensus.cc:1226] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:04.008176 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:04.014804 16632 raft_consensus.cc:3037] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:04.015237 17785 raft_consensus.cc:1261] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260810 18:52:03.969378 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:04.016593 16633 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:04.019327 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:04.020848 16742 raft_consensus.cc:1226] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:04.021008 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:04.022444 16749 raft_consensus.cc:1226] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 2. Current term is 3. Ops: []
I20260810 18:52:03.974224 19713 raft_consensus.cc:2726] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260810 18:52:04.008690 16448 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:04.024890 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:04.025286 16636 raft_consensus.cc:2454] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:04.026031 16504 raft_consensus.cc:2396] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260810 18:52:04.027431 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:03.988369 19646 raft_consensus.cc:482] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:04.031431 16695 leader_election.cc:302] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:04.034134 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
W20260810 18:52:03.975808 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 51494.6us
I20260810 18:52:04.042855 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:04.039985 16495 raft_consensus.cc:1261] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260810 18:52:04.042289 16695 leader_election.cc:302] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:04.006394 19373 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:04.051532 16632 raft_consensus.cc:2454] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
I20260810 18:52:04.026120 19718 raft_consensus.cc:686] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:04.031530 19660 raft_consensus.cc:3032] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:52:04.061112 16742 raft_consensus.cc:1226] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:04.049134 19473 raft_consensus.cc:2726] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:52:04.012981 19480 consensus_queue.cc:1046] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260810 18:52:04.051438 19504 raft_consensus.cc:2726] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:52:04.078372 19244 consensus_queue.cc:1046] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20260810 18:52:04.064838 19495 raft_consensus.cc:2726] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260810 18:52:04.070236 19380 raft_consensus.cc:482] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:04.065419 19660 raft_consensus.cc:729] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:04.041226 19646 raft_consensus.cc:504] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:04.100279 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:04.070716 19684 consensus_queue.cc:1046] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.030s
I20260810 18:52:04.102753 16740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1493) took 1386 ms. Trace:
I20260810 18:52:04.107445 16740 rpcz_store.cc:276] 0810 18:52:02.716632 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:03.383766 (+667134us) service_pool.cc:227] Handling call
0810 18:52:03.384042 (+ 276us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:03.791708 (+407666us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:04.102514 (+310806us) spinlock_profiling.cc:243] Waited 406 ms on lock 0x7b54003bf454. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:04.102691 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":933203328}
I20260810 18:52:04.091547 19723 raft_consensus.cc:482] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:04.106277 16633 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:04.110582 19495 raft_consensus.cc:3032] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:52:04.065662 18728 consensus_queue.cc:1046] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260810 18:52:04.091452 19595 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:04.069759 19587 consensus_queue.cc:1046] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
W20260810 18:52:04.118724 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 54790.4us
I20260810 18:52:04.067978 17727 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:04.126693 16695 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:04.128497 16695 leader_election.cc:302] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:04.111052 19495 raft_consensus.cc:729] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:04.105355 19504 consensus_queue.cc:1046] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260810 18:52:04.130522 19391 raft_consensus.cc:482] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:04.135293 19729 log.cc:924] Time spent T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.116s user 0.001s sys 0.000s
I20260810 18:52:04.130312 18808 leader_election.cc:288] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:04.131611 19323 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:04.108098 19380 raft_consensus.cc:504] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:04.131136 19373 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:04.116848 17775 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55602 (request call id 309) took 1799 ms. Trace:
I20260810 18:52:04.115579 19723 raft_consensus.cc:504] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:04.155445 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 99293us
I20260810 18:52:04.118762 19392 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:04.148268 19635 raft_consensus.cc:2781] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:52:04.120893 19595 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:04.161849 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:04.166280 16742 raft_consensus.cc:2379] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 3: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
W20260810 18:52:04.151002 19732 log.cc:924] Time spent T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
W20260810 18:52:04.160640 19724 log.cc:924] Time spent T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.113s user 0.000s sys 0.001s
W20260810 18:52:04.153039 19725 log.cc:924] Time spent T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.063s user 0.001s sys 0.000s
I20260810 18:52:04.160706 19495 consensus_queue.cc:260] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:04.158460 17775 rpcz_store.cc:276] 0810 18:52:02.316069 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:02.321510 (+ 5441us) service_pool.cc:227] Handling call
0810 18:52:04.100168 (+1778658us) spinlock_profiling.cc:243] Waited 19 ms on lock 0x7b5800005498. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519abce 000000000034aea9 00007f1aecfc6a30 00007f1aed08c7b4 00007f1aed08c821 00007f1aed03c1b0 00007f1aed03ba94 00007f1ae6723c04 00007f1ae6723b92
0810 18:52:04.115177 (+ 15009us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":43731712}
I20260810 18:52:04.177436 19495 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:04.181298 19496 raft_consensus.cc:504] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:04.144918 18903 raft_consensus.cc:504] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:04.183414 19496 leader_election.cc:288] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:04.183954 19496 raft_consensus.cc:3032] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:52:04.184264 19496 raft_consensus.cc:729] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:04.184680 18903 leader_election.cc:288] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:04.185016 19496 consensus_queue.cc:260] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:52:04.186479 16750 raft_consensus.cc:1226] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:04.149824 19543 raft_consensus.cc:504] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:04.189033 19543 leader_election.cc:288] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:04.182902 19731 log.cc:924] Time spent T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.151s user 0.001s sys 0.000s
I20260810 18:52:04.154395 19554 raft_consensus.cc:2726] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:52:04.089570 18948 consensus_queue.cc:237] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:04.172890 16740 raft_consensus.cc:1261] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260810 18:52:04.166957 19373 raft_consensus.cc:3032] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:52:04.191661 16636 raft_consensus.cc:1226] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 2. Current term is 3. Ops: []
I20260810 18:52:04.192632 17727 leader_election.cc:302] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:04.151130 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:04.118297 19718 consensus_queue.cc:237] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:04.169965 19635 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:04.154593 19391 raft_consensus.cc:504] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:04.166026 18808 consensus_queue.cc:1035] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20260810 18:52:04.235754 19504 consensus_queue.cc:1035] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260810 18:52:04.236989 19738 log.cc:924] Time spent T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.103s user 0.000s sys 0.001s
I20260810 18:52:04.135094 19660 consensus_queue.cc:260] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:04.215091 19373 raft_consensus.cc:729] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:04.252974 19708 raft_consensus.cc:504] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:04.257416 16626 raft_consensus.cc:1226] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260810 18:52:04.242003 19577 raft_consensus.cc:3037] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:04.243880 19635 raft_consensus.cc:3037] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
W20260810 18:52:04.274688 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 248304us
I20260810 18:52:04.278487 16448 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:04.255376 19741 raft_consensus.cc:3032] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:52:04.249301 19291 consensus_queue.cc:1035] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
W20260810 18:52:04.313593 19648 log.cc:924] Time spent T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 1.169s user 0.003s sys 0.005s
I20260810 18:52:04.314978 16506 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1385) took 1681 ms. Trace:
I20260810 18:52:04.316038 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:04.317029 16636 raft_consensus.cc:2379] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 3: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
W20260810 18:52:04.318607 17783 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55602 (request call id 310) took 1997 ms (client timeout 2642 ms). Trace:
I20260810 18:52:04.319595 16745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1474) took 2031 ms. Trace:
I20260810 18:52:04.320459 19380 leader_election.cc:288] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:04.321676 17786 raft_consensus.cc:1226] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: []
W20260810 18:52:04.322487 17783 rpcz_store.cc:269] 0810 18:52:02.321289 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:02.329261 (+ 7972us) service_pool.cc:227] Handling call
0810 18:52:03.126808 (+797547us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b5800005498. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed03c1b0 00007f1aed03ba94 00007f1ae6723c04 00007f1ae6723b92
0810 18:52:04.318376 (+1191568us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b54006b1554. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:04.318536 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":61571328}
I20260810 18:52:04.320065 16745 rpcz_store.cc:276] 0810 18:52:02.287713 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:02.289116 (+ 1403us) service_pool.cc:227] Handling call
0810 18:52:02.289401 (+ 285us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:03.277845 (+988444us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:04.319319 (+1041474us) spinlock_profiling.cc:243] Waited 988 ms on lock 0x7b5400156654. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:04.319520 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2271907712}
I20260810 18:52:04.315407 16506 rpcz_store.cc:276] 0810 18:52:02.633540 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:02.661745 (+ 28205us) service_pool.cc:227] Handling call
0810 18:52:02.662057 (+ 312us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:02.662369 (+ 312us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:02.662396 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:02.662798 (+ 402us) log.cc:842] Serialized 26 byte log entry
0810 18:52:02.990859 (+328061us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:02.990922 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:02.991021 (+ 99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:04.313273 (+1322252us) spinlock_profiling.cc:243] Waited 62.2 ms on lock 0x7b240018dcb8. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae502acf4 00007f1ae5028dfb 00007f1ae50b7559 00007f1ae50b71e5 00007f1ae50b682d 00007f1ae7d6546e 00007f1ae7d7cc0c 00007f1ae7d7bf36 00007f1ae7d7bb58 00007f1ae7d85551
0810 18:52:04.314641 (+ 1368us) raft_consensus.cc:1671] finished
0810 18:52:04.314663 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:04.314915 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2168,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":8027,"spinlock_wait_cycles":142988032,"thread_start_us":327649,"threads_started":1,"wal-append.queue_time_us":481285}
I20260810 18:52:04.324913 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:04.325779 16749 raft_consensus.cc:2454] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 3.
I20260810 18:52:04.328183 19392 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:04.329499 17727 leader_election.cc:302] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:04.329773 19595 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:04.335486 19391 leader_election.cc:288] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:04.335721 19723 leader_election.cc:288] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:04.336241 19697 raft_consensus.cc:3032] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:52:04.336879 19697 raft_consensus.cc:729] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:04.268031 19646 leader_election.cc:288] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:04.272011 19524 ts_tablet_manager.cc:1918] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260810 18:52:04.350152 19606 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:04.330914 19380 raft_consensus.cc:2781] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:04.289706 19708 leader_election.cc:288] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:04.336284 19373 consensus_queue.cc:260] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:04.379585 19373 raft_consensus.cc:3037] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:04.392586 16342 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55260 (request call id 122) took 8100 ms. Trace:
I20260810 18:52:04.303807 19741 raft_consensus.cc:729] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:04.339362 19752 raft_consensus.cc:2781] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:04.360473 19524 log.cc:1196] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb: Deleting WAL directory at /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-1-root/wals/7ef7c65533724e678b75830002906e2c
I20260810 18:52:04.361033 19606 raft_consensus.cc:3037] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:04.405097 19391 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:04.412822 17778 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55586 (request call id 284) took 1758 ms (client timeout 1826 ms). Trace:
I20260810 18:52:04.415868 16752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1480) took 2124 ms. Trace:
I20260810 18:52:04.427453 17770 raft_consensus.cc:1226] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 2. Current term is 3. Ops: []
I20260810 18:52:04.470261 19496 raft_consensus.cc:3037] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:52:04.368026 19380 raft_consensus.cc:482] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:04.442656 16752 rpcz_store.cc:276] 0810 18:52:02.291633 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:02.293129 (+ 1496us) service_pool.cc:227] Handling call
0810 18:52:02.293386 (+ 257us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:03.216974 (+923588us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:04.415548 (+1198574us) spinlock_profiling.cc:243] Waited 826 ms on lock 0x7b54002ed1d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:04.415790 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1898999552}
I20260810 18:52:04.444778 19391 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:04.463110 19684 raft_consensus.cc:3032] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:52:04.495734 19684 raft_consensus.cc:729] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:04.497143 19684 consensus_queue.cc:260] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:04.466848 19351 raft_consensus.cc:3032] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:52:04.468492 19543 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:04.499698 19351 raft_consensus.cc:729] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:04.499920 19543 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:04.500679 19351 consensus_queue.cc:260] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:04.501833 19351 raft_consensus.cc:3037] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:04.502333 19543 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:04.431838 19657 raft_consensus.cc:3032] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:52:04.502995 19657 raft_consensus.cc:729] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:04.506968 17770 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55602 (request call id 314) took 1880 ms. Trace:
I20260810 18:52:04.507478 17770 rpcz_store.cc:276] 0810 18:52:02.626476 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:03.013633 (+387157us) service_pool.cc:227] Handling call
0810 18:52:04.427138 (+1413505us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:04.482916 (+ 55778us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:04.506648 (+ 23732us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b540072e550. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:04.506858 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3240628608}
W20260810 18:52:04.479735 17778 rpcz_store.cc:269] 0810 18:52:02.654574 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:02.656033 (+ 1459us) service_pool.cc:227] Handling call
0810 18:52:04.412570 (+1756537us) spinlock_profiling.cc:243] Waited 856 ms on lock 0x7b540072e554. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:04.412744 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1968812288}
I20260810 18:52:04.513147 19660 raft_consensus.cc:3037] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:04.401320 16500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56468 (request call id 145) took 1192 ms. Trace:
I20260810 18:52:04.338061 19697 consensus_queue.cc:260] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:04.530296 19391 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:04.531252 19741 consensus_queue.cc:260] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:04.557077 19741 raft_consensus.cc:3037] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:04.490634 19380 raft_consensus.cc:3037] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:04.542444 19524 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53868 (request call id 959) took 4220 ms. Trace:
I20260810 18:52:04.521502 16500 rpcz_store.cc:276] 0810 18:52:03.208833 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:03.216831 (+ 7998us) service_pool.cc:227] Handling call
0810 18:52:04.400275 (+1183444us) spinlock_profiling.cc:243] Waited 972 us on lock 0x7b5800001898. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed0502b0 00007f1aed03b7f9 00007f1aed03b2a2 00007f1ae6721684
0810 18:52:04.400569 (+ 294us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:04.400821 (+ 252us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:04.400865 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:04.400931 (+ 66us) raft_consensus.cc:1631] Marking committed up to 7
0810 18:52:04.400981 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:04.401067 (+ 86us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:04.401233 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2235136}
I20260810 18:52:04.498946 19684 raft_consensus.cc:3037] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:04.566334 19495 raft_consensus.cc:482] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:04.576809 19495 raft_consensus.cc:504] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:04.567020 19593 raft_consensus.cc:482] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:04.579813 19593 raft_consensus.cc:504] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:04.582698 19593 leader_election.cc:288] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:04.585250 19495 leader_election.cc:288] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:04.555455 19697 raft_consensus.cc:3037] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:04.574834 19524 rpcz_store.cc:276] 0810 18:52:00.321912 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:00.343176 (+ 21264us) service_pool.cc:227] Handling call
0810 18:52:01.198567 (+855391us) ts_tablet_manager.cc:1145] Deleting tablet 7ef7c65533724e678b75830002906e2c
0810 18:52:01.201807 (+ 3240us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 18:52:03.959868 (+2758061us) tablet_metadata.cc:641] Metadata flushed
0810 18:52:04.254599 (+294731us) tablet_metadata.cc:641] Metadata flushed
0810 18:52:04.542305 (+287706us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":491548,"spinlock_wait_cycles":1893632,"tablet-delete.queue_time_us":796492,"thread_start_us":791111,"threads_started":1}
W20260810 18:52:04.637248 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 500020us
I20260810 18:52:04.647159 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:52:04.678617 16632 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.612s user 0.017s sys 0.036s
W20260810 18:52:04.679435 16632 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.613s user 0.017s sys 0.036s
W20260810 18:52:04.710366 16450 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.636s user 0.009s sys 0.043s
W20260810 18:52:04.717151 16450 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.643s user 0.009s sys 0.043s
W20260810 18:52:04.717576 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 653056us
W20260810 18:52:04.762233 19168 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.611s user 0.009s sys 0.033s
W20260810 18:52:04.762921 19168 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.615s user 0.009s sys 0.034s
W20260810 18:52:04.787881 16345 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37026 (request call id 225) took 13702 ms (client timeout 15000 ms). Trace:
I20260810 18:52:04.791133 19589 raft_consensus.cc:3032] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Stepping down as leader of term 3
W20260810 18:52:04.792374 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 73980us
I20260810 18:52:04.393553 16342 rpcz_store.cc:276] 0810 18:51:56.364347 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:56.367732 (+ 3385us) service_pool.cc:227] Handling call
0810 18:52:04.392496 (+8024764us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 18:51:58.291847 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 18:51:58.489049 (+197202us) write_op.cc:432] Acquiring schema lock in shared mode
0810 18:51:58.491907 (+ 2858us) write_op.cc:435] Acquired schema lock
0810 18:51:58.491936 (+ 29us) tablet.cc:660] Decoding operations
0810 18:51:58.716889 (+224953us) write_op.cc:620] Acquiring the partition lock for write op
0810 18:51:58.719240 (+ 2351us) write_op.cc:641] Partition lock acquired for write op
0810 18:51:58.720815 (+ 1575us) tablet.cc:683] Acquiring locks for 5 operations
0810 18:51:58.810590 (+ 89775us) tablet.cc:699] Row locks acquired
0810 18:51:58.810649 (+ 59us) write_op.cc:260] PREPARE: finished
0810 18:51:58.810979 (+ 330us) write_op.cc:270] Start()
0810 18:51:58.815237 (+ 4258us) write_op.cc:276] Timestamp: P: 1786387918810894 usec, L: 0
0810 18:51:58.820616 (+ 5379us) op_driver.cc:331] REPLICATION: starting
0810 18:51:59.243775 (+423159us) log.cc:842] Serialized 2035 byte log entry
0810 18:52:01.392227 (+2148452us) op_driver.cc:444] REPLICATION: finished
0810 18:52:01.967562 (+575335us) write_op.cc:301] APPLY: starting
0810 18:52:01.987357 (+ 19795us) tablet.cc:1365] starting BulkCheckPresence
0810 18:52:02.033546 (+ 46189us) tablet.cc:1368] finished BulkCheckPresence
0810 18:52:02.033698 (+ 152us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 18:52:02.085625 (+ 51927us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 18:52:02.135344 (+ 49719us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0810 18:52:02.142193 (+ 6849us) write_op.cc:312] APPLY: finished
0810 18:52:02.197945 (+ 55752us) log.cc:842] Serialized 47 byte log entry
0810 18:52:02.265849 (+ 67904us) write_op.cc:489] Releasing partition, row and schema locks
0810 18:52:02.285691 (+ 19842us) write_op.cc:454] Released schema lock
0810 18:52:02.289509 (+ 3818us) write_op.cc:341] FINISH: Updating metrics
0810 18:52:02.387916 (+ 98407us) write_op.cc:489] Releasing partition, row and schema locks
0810 18:52:02.391188 (+ 3272us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":17,"child_traces":[["op",{"apply.queue_time_us":532570,"apply.run_cpu_time_us":4685,"apply.run_wall_time_us":468888,"num_ops":5,"prepare.queue_time_us":314132,"prepare.run_cpu_time_us":24590,"prepare.run_wall_time_us":2107602,"raft.queue_time_us":528605,"raft.run_cpu_time_us":15801,"raft.run_wall_time_us":428284,"replication_time_us":2556837,"thread_start_us":1846049,"threads_started":4,"wal-append.queue_time_us":1351627,"wal-append.run_cpu_time_us":41548,"wal-append.run_wall_time_us":2673119}]]}
I20260810 18:52:04.803704 16695 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
W20260810 18:52:04.832749 19584 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.599s user 0.005s sys 0.008s
W20260810 18:52:04.833276 19584 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.600s user 0.005s sys 0.008s
W20260810 18:52:04.835139 16448 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.020s sys 0.010s
W20260810 18:52:04.836403 16448 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.534s user 0.021s sys 0.011s
W20260810 18:52:04.837190 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 548666us
W20260810 18:52:04.838186 16702 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.684s user 0.043s sys 0.030s
W20260810 18:52:04.838558 16702 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.684s user 0.043s sys 0.030s
I20260810 18:52:04.843819 16748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 141) took 2203 ms. Trace:
W20260810 18:52:04.798520 19471 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.000s sys 0.004s
W20260810 18:52:04.794318 16573 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.841s user 0.020s sys 0.027s
W20260810 18:52:04.793958 16631 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.760s user 0.015s sys 0.020s
W20260810 18:52:04.829063 19470 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.597s user 0.005s sys 0.006s
W20260810 18:52:04.828692 16345 rpcz_store.cc:269] 0810 18:51:51.085734 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:51:51.085905 (+ 171us) service_pool.cc:227] Handling call
0810 18:52:04.787672 (+13701767us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":944273,"reported_tablets":7}
W20260810 18:52:04.833557 19544 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.453s user 0.012s sys 0.011s
I20260810 18:52:04.791664 19752 raft_consensus.cc:686] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 6 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
W20260810 18:52:04.864138 16633 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.716s user 0.001s sys 0.022s
W20260810 18:52:04.864470 16633 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.716s user 0.001s sys 0.022s
W20260810 18:52:04.851097 16575 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.898s user 0.019s sys 0.023s
I20260810 18:52:04.822777 16344 catalog_manager.cc:5495] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 1 to 2, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 2 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
W20260810 18:52:04.868124 16575 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.915s user 0.019s sys 0.023s
W20260810 18:52:04.868944 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 926409us
I20260810 18:52:04.868768 16344 catalog_manager.cc:5495] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 2 to 4, leader changed from 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 4 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260810 18:52:04.870098 16344 catalog_manager.cc:5495] T 0c3053ea9475438bab46cba64f4cc2fa P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 1 to 2, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 2 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:52:04.870582 16575 leader_election.cc:302] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:04.854866 16631 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.819s user 0.015s sys 0.020s
W20260810 18:52:04.852622 16573 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.900s user 0.020s sys 0.028s
W20260810 18:52:04.882532 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 933778us
I20260810 18:52:04.884675 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:04.886128 16573 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:04.886221 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:04.886830 17787 raft_consensus.cc:3037] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:04.826975 19589 raft_consensus.cc:729] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:04.872103 16344 catalog_manager.cc:5495] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 1 to 2, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 2 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
W20260810 18:52:04.830188 16621 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.006s sys 0.003s
I20260810 18:52:04.980764 16344 catalog_manager.cc:5495] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 3 to 5, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 5 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:52:04.835992 18903 raft_consensus.cc:482] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
W20260810 18:52:04.853185 19471 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.624s user 0.000s sys 0.004s
W20260810 18:52:04.898176 17724 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.656s user 0.019s sys 0.017s
I20260810 18:52:04.988281 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:04.988665 17724 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.746s user 0.031s sys 0.033s
I20260810 18:52:04.988970 16507 raft_consensus.cc:2454] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
W20260810 18:52:04.989101 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 761166us
W20260810 18:52:04.867221 19544 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.486s user 0.012s sys 0.011s
W20260810 18:52:04.956863 16701 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.508s (SENT) [suppressed 40 similar messages]
I20260810 18:52:04.959662 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:52:04.958514 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 110142us
I20260810 18:52:04.970954 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
W20260810 18:52:04.970750 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 100314us
I20260810 18:52:04.971786 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:04.972741 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:04.975445 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:04.854262 19758 raft_consensus.cc:2781] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election won for term 3
W20260810 18:52:04.976670 16624 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.814s user 0.090s sys 0.093s
W20260810 18:52:04.997184 16624 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.835s user 0.090s sys 0.093s
I20260810 18:52:04.846864 19552 raft_consensus.cc:2726] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260810 18:52:04.953303 16573 raft_consensus.cc:2670] T 7ef7c65533724e678b75830002906e2c P 0397d760589d40d3b92ef141cdfc2dbb: unable to run election callback: Service unavailable: Thread pool token was shut down
W20260810 18:52:05.000221 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 114186us
I20260810 18:52:05.000893 16573 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:04.962360 19778 log.cc:924] Time spent T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.171s user 0.000s sys 0.001s
I20260810 18:52:04.982491 16344 catalog_manager.cc:5495] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 1 to 2, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 2 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
W20260810 18:52:04.983842 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 122053us
I20260810 18:52:04.985581 16501 raft_consensus.cc:1226] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260810 18:52:04.862576 19470 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.631s user 0.005s sys 0.006s
W20260810 18:52:04.855407 19099 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.744s user 0.022s sys 0.057s
I20260810 18:52:04.997692 16575 leader_election.cc:302] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:04.872452 19561 consensus_queue.cc:1046] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.110s
I20260810 18:52:04.955716 19749 raft_consensus.cc:482] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
W20260810 18:52:04.981822 16621 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.740s user 0.007s sys 0.003s
I20260810 18:52:05.052507 16573 leader_election.cc:302] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:05.067617 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 70014.5us
I20260810 18:52:05.068529 16575 leader_election.cc:302] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:05.069796 16575 leader_election.cc:302] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:05.051286 16344 catalog_manager.cc:5495] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 2 to 3, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 3 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:52:04.975170 19366 raft_consensus.cc:3037] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:04.989933 19495 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:05.049597 19735 log.cc:924] Time spent T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
W20260810 18:52:05.055137 19099 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.944s user 0.025s sys 0.062s
I20260810 18:52:05.074777 16344 catalog_manager.cc:5495] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 1 to 2, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 2 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:52:05.062506 19543 raft_consensus.cc:2726] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:04.874310 19728 consensus_queue.cc:1046] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.108s
I20260810 18:52:04.990976 19748 consensus_queue.cc:1046] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.085s
I20260810 18:52:05.056515 19758 raft_consensus.cc:686] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:05.055570 19593 raft_consensus.cc:482] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:05.116586 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 46039.1us
I20260810 18:52:05.118283 16344 catalog_manager.cc:5495] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 2 to 4, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 4 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:52:04.999372 18903 raft_consensus.cc:504] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:05.081234 19391 raft_consensus.cc:2726] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260810 18:52:05.077235 19791 log.cc:924] Time spent T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.090s user 0.001s sys 0.001s
I20260810 18:52:05.076256 19470 raft_consensus.cc:2726] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:52:05.115639 19755 raft_consensus.cc:2726] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:05.152064 19543 raft_consensus.cc:2781] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:05.144680 19544 raft_consensus.cc:482] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:05.155774 19790 log.cc:924] Time spent T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20260810 18:52:05.122313 19279 consensus_queue.cc:1046] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
W20260810 18:52:05.189875 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 77511us
W20260810 18:52:05.194718 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 113503us
I20260810 18:52:05.083772 19495 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:05.123481 19446 consensus_queue.cc:1046] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
W20260810 18:52:05.172209 19635 consensus_meta.cc:287] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.885s user 0.011s sys 0.001s
I20260810 18:52:05.073931 19749 raft_consensus.cc:504] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:05.202696 16747 raft_consensus.cc:1261] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260810 18:52:05.184919 19577 consensus_meta.cc:287] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.913s user 0.011s sys 0.000s
I20260810 18:52:05.075788 19752 consensus_queue.cc:237] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:05.208310 16742 raft_consensus.cc:1226] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:05.212648 16742 raft_consensus.cc:1226] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 2. Current term is 3. Ops: []
I20260810 18:52:05.151723 19480 consensus_queue.cc:1046] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20260810 18:52:05.215320 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:05.120703 19589 consensus_queue.cc:260] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:05.216063 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:05.186345 19543 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:05.216356 16508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:52330 (request call id 799) took 1180 ms. Trace:
I20260810 18:52:05.221565 16750 raft_consensus.cc:2454] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:05.221730 16508 rpcz_store.cc:276] 0810 18:52:04.027404 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:04.027544 (+ 140us) service_pool.cc:227] Handling call
0810 18:52:05.206606 (+1179062us) spinlock_profiling.cc:243] Waited 819 ms on lock 0x7b54000f7154. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807faf0 00007f1ae8102f2a 00007f1aed03fc6e 00007f1ae6736807 00007f1ae6736792 00007f1ae67366c1 00007f1ae673663c 00007f1ae67358f2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:05.207361 (+ 755us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1883970816}
W20260810 18:52:05.215657 19793 log.cc:924] Time spent T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.063s user 0.000s sys 0.001s
W20260810 18:52:05.206362 19782 log.cc:924] Time spent T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.135s user 0.000s sys 0.000s
I20260810 18:52:05.159369 19593 raft_consensus.cc:504] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:05.192783 19391 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
W20260810 18:52:05.223798 19802 log.cc:924] Time spent T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.148s user 0.000s sys 0.001s
I20260810 18:52:05.201992 19561 raft_consensus.cc:3032] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:52:05.206550 19764 raft_consensus.cc:482] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:05.242282 17787 raft_consensus.cc:2454] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:05.217446 19577 raft_consensus.cc:2726] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:52:05.254465 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:05.257355 16507 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:05.195130 18948 consensus_queue.cc:1046] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260810 18:52:05.226577 19543 raft_consensus.cc:3037] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:05.194669 19544 raft_consensus.cc:504] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:05.260365 16695 leader_election.cc:302] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:05.241235 19806 log.cc:924] Time spent T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
W20260810 18:52:05.267100 19794 log.cc:924] Time spent T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.111s user 0.000s sys 0.000s
W20260810 18:52:05.241379 19779 log.cc:924] Time spent T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20260810 18:52:05.237536 19589 raft_consensus.cc:3037] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:05.212174 19635 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:05.250738 19561 raft_consensus.cc:729] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:05.253366 19787 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:05.271152 19789 log.cc:924] Time spent T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.081s user 0.000s sys 0.001s
I20260810 18:52:05.253422 19099 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:05.270534 16621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1477) took 1290 ms. Trace:
W20260810 18:52:05.286321 19660 consensus_meta.cc:287] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.772s user 0.013s sys 0.000s
I20260810 18:52:05.263111 16746 raft_consensus.cc:2454] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:05.278888 19595 raft_consensus.cc:2781] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:05.263541 19279 consensus_queue.cc:1046] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20260810 18:52:05.247488 19391 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:05.372766 19495 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:05.325644 19496 consensus_meta.cc:287] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.832s user 0.008s sys 0.004s
I20260810 18:52:05.260635 19764 raft_consensus.cc:504] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:05.373042 19544 leader_election.cc:288] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:05.296924 19289 consensus_queue.cc:1046] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.071s
I20260810 18:52:05.227198 19758 consensus_queue.cc:237] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:05.383941 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 300781us
I20260810 18:52:05.328240 19787 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:05.379468 19593 leader_election.cc:288] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:05.469115 19289 raft_consensus.cc:3032] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:52:05.469487 19289 raft_consensus.cc:729] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:05.326220 19099 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:05.470336 19748 raft_consensus.cc:3032] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:52:05.470784 19748 raft_consensus.cc:729] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:05.470963 19635 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:05.471604 19099 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:05.472846 19606 consensus_meta.cc:287] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.982s user 0.008s sys 0.004s
I20260810 18:52:05.471630 19748 consensus_queue.cc:260] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:05.473484 19606 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:05.476897 19764 leader_election.cc:288] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:05.441321 17786 raft_consensus.cc:1226] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 2. Current term is 3. Ops: []
I20260810 18:52:05.474123 19748 raft_consensus.cc:3037] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:05.478889 19740 raft_consensus.cc:3032] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Stepping down as leader of term 4
I20260810 18:52:05.479418 19740 raft_consensus.cc:729] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:05.480383 19740 consensus_queue.cc:260] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:05.481966 19740 raft_consensus.cc:3037] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:05.445034 18903 leader_election.cc:288] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:05.439917 19380 consensus_meta.cc:287] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.866s user 0.007s sys 0.010s
W20260810 18:52:05.492326 19807 log.cc:924] Time spent T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.284s user 0.000s sys 0.001s
I20260810 18:52:05.447213 19657 consensus_queue.cc:260] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:05.493244 19657 raft_consensus.cc:3037] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
W20260810 18:52:05.493803 19776 log.cc:924] Time spent T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.198s user 0.001s sys 0.003s
W20260810 18:52:05.494221 19772 log.cc:924] Time spent T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.238s user 0.001s sys 0.004s
I20260810 18:52:05.494529 19606 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:05.449126 19168 raft_consensus.cc:3032] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:52:05.495216 19168 raft_consensus.cc:729] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:05.496073 19168 consensus_queue.cc:260] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:05.497347 19168 raft_consensus.cc:3037] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:05.470391 19787 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:05.446115 19561 consensus_queue.cc:260] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:05.456249 19391 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:05.458079 19658 raft_consensus.cc:3032] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:52:05.458668 19639 raft_consensus.cc:3032] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Stepping down as leader of term 3
W20260810 18:52:05.459656 19809 log.cc:924] Time spent T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20260810 18:52:05.389570 19595 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260810 18:52:05.492337 19697 consensus_meta.cc:287] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.899s user 0.011s sys 0.000s
W20260810 18:52:05.442353 19684 consensus_meta.cc:287] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.856s user 0.009s sys 0.003s
I20260810 18:52:05.444460 19749 leader_election.cc:288] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:05.474072 19799 log.cc:924] Time spent T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.183s user 0.000s sys 0.004s
W20260810 18:52:05.493203 19741 consensus_meta.cc:287] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.923s user 0.004s sys 0.006s
W20260810 18:52:05.514345 19805 log.cc:924] Time spent T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.273s user 0.003s sys 0.003s
W20260810 18:52:05.545109 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 240544us
I20260810 18:52:05.549034 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:05.553385 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
W20260810 18:52:05.545763 16504 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.004s sys 0.003s
I20260810 18:52:05.554658 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:05.553653 16495 raft_consensus.cc:2454] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
W20260810 18:52:05.555321 16504 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.555s user 0.004s sys 0.003s
W20260810 18:52:05.559521 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 247556us
I20260810 18:52:05.561100 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:05.307951 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 255510us
I20260810 18:52:05.569681 16507 raft_consensus.cc:1226] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:05.571427 16573 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:52:05.571265 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
W20260810 18:52:05.575425 16575 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.002s sys 0.006s
W20260810 18:52:05.576041 16575 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.506s user 0.002s sys 0.006s
I20260810 18:52:05.578606 16450 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:05.323335 16621 rpcz_store.cc:276] 0810 18:52:03.973202 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:04.239679 (+266477us) service_pool.cc:227] Handling call
0810 18:52:04.240122 (+ 443us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:04.240410 (+ 288us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:04.240445 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:04.241012 (+ 567us) log.cc:842] Serialized 26 byte log entry
0810 18:52:05.076058 (+835046us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:52:05.081136 (+ 5078us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:05.115214 (+ 34078us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:05.235956 (+120742us) raft_consensus.cc:1671] finished
0810 18:52:05.235984 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:05.263628 (+ 27644us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":825674,"threads_started":1,"wal-append.queue_time_us":824678}
W20260810 18:52:05.578302 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 508515us
I20260810 18:52:05.587833 16622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1482) took 1611 ms. Trace:
I20260810 18:52:05.591717 16624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1476) took 1628 ms. Trace:
I20260810 18:52:05.594758 16575 leader_election.cc:302] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:05.595321 16624 rpcz_store.cc:276] 0810 18:52:03.972599 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:04.114643 (+142044us) service_pool.cc:227] Handling call
0810 18:52:04.127446 (+ 12803us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:04.127723 (+ 277us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:04.127759 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:04.133801 (+ 6042us) log.cc:842] Serialized 26 byte log entry
0810 18:52:04.997720 (+863919us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:52:04.997754 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:04.997815 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:05.493693 (+495878us) spinlock_profiling.cc:243] Waited 70.4 ms on lock 0x7b44000232cc. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae50e4a95 00007f1ae50e54b2 00007f1ae7d8d511 00007f1ae7d88e1a 00007f1ae7d7cbee 00007f1ae7d7bf36 00007f1ae7d7bb58 00007f1ae7d85551 00007f1ae7d85509 00007f1ae7d85499
0810 18:52:05.591019 (+ 97326us) raft_consensus.cc:1671] finished
0810 18:52:05.591085 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:05.591505 (+ 420us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":161990912,"thread_start_us":835092,"threads_started":1,"wal-append.queue_time_us":1108077}
I20260810 18:52:05.593159 16622 rpcz_store.cc:276] 0810 18:52:03.976543 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:04.353740 (+377197us) service_pool.cc:227] Handling call
0810 18:52:04.354115 (+ 375us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:04.354537 (+ 422us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:04.354564 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:04.358961 (+ 4397us) log.cc:842] Serialized 26 byte log entry
0810 18:52:04.830872 (+471911us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:04.830982 (+ 110us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:04.831127 (+ 145us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:05.504686 (+673559us) spinlock_profiling.cc:243] Waited 85.9 ms on lock 0x7b44000232cc. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae50e4a95 00007f1ae50e54b2 00007f1ae7d8d511 00007f1ae7d88e1a 00007f1ae7d7cbee 00007f1ae7d7bf36 00007f1ae7d7bb58 00007f1ae7d85551 00007f1ae7d85509 00007f1ae7d85499
0810 18:52:05.586983 (+ 82297us) raft_consensus.cc:1671] finished
0810 18:52:05.587040 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:05.587543 (+ 503us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":91,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":130,"spinlock_wait_cycles":197584128,"thread_start_us":471388,"threads_started":1,"wal-append.queue_time_us":881734}
W20260810 18:52:05.599472 16701 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.002s sys 0.010s
I20260810 18:52:05.599519 16633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1475) took 1636 ms. Trace:
W20260810 18:52:05.600137 16701 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.588s user 0.003s sys 0.010s
W20260810 18:52:05.600871 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 644014us
I20260810 18:52:05.600193 16633 rpcz_store.cc:276] 0810 18:52:03.962895 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:04.110357 (+147462us) service_pool.cc:227] Handling call
0810 18:52:04.111996 (+ 1639us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:04.112261 (+ 265us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:04.112301 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:04.117001 (+ 4700us) log.cc:842] Serialized 26 byte log entry
0810 18:52:04.864794 (+747793us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:04.977430 (+112636us) log.cc:842] Serialized 14 byte log entry
0810 18:52:04.997258 (+ 19828us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:04.997332 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:05.456810 (+459478us) spinlock_profiling.cc:243] Waited 12.8 ms on lock 0x7b180031bee0. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1aee0718b1 00007f1aee07e661 00007f1ae51d2764 00007f1ae51d5b11 00007f1ae51d5ac9 00007f1ae51d5a59 00007f1ae51d5a21 00007f1ae51d4d1d 00007f1aee436204 00007f1aee436039 00007f1ae51b4dc6
0810 18:52:05.598897 (+142087us) raft_consensus.cc:1671] finished
0810 18:52:05.599021 (+ 124us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:05.599330 (+ 309us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19578,"prepare.queue_time_us":115336,"prepare.run_cpu_time_us":807,"prepare.run_wall_time_us":5494,"raft.queue_time_us":19723,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":53177,"spinlock_wait_cycles":29383296,"thread_start_us":716701,"threads_started":1,"wal-append.queue_time_us":949815}
W20260810 18:52:05.617846 16701 leader_election.cc:334] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.676s (SENT)
I20260810 18:52:05.370905 16740 raft_consensus.cc:1226] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 2. Current term is 3. Ops: []
I20260810 18:52:05.620024 16745 raft_consensus.cc:1261] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260810 18:52:04.844432 16748 rpcz_store.cc:276] 0810 18:52:02.640654 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:02.861753 (+221099us) service_pool.cc:227] Handling call
0810 18:52:02.917065 (+ 55312us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:02.917250 (+ 185us) raft_consensus.cc:1520] Early marking committed up to index 5
0810 18:52:02.957526 (+ 40276us) log.cc:842] Serialized 14 byte log entry
0810 18:52:04.438331 (+1480805us) spinlock_profiling.cc:243] Waited 20.1 ms on lock 0x7b1800223c00. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1aee0718b1 00007f1aee07e661 00007f1ae51b4345 00007f1aedfeba39 00007f1ae51d18bb 00007f1ae51cfcbf 00007f1ae51d1e22 00007f1ae7d7b5bb 00007f1ae7d81063 00007f1ae7d8157b 00007f1ae810488a
0810 18:52:04.843332 (+405001us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:04.843423 (+ 91us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:04.843492 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:04.843615 (+ 123us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:04.843759 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":42251,"prepare.queue_time_us":40292,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":3959,"raft.queue_time_us":44932,"spinlock_wait_cycles":46133120,"thread_start_us":361620,"threads_started":1}
I20260810 18:52:05.378043 16749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 163) took 1037 ms. Trace:
I20260810 18:52:05.624347 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:05.629480 16450 leader_election.cc:302] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:05.639962 16575 leader_election.cc:398] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Vote denied by peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171) with higher term. Message: Invalid argument: T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 3. Current term is 4.
W20260810 18:52:05.652305 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 189301us
I20260810 18:52:05.656418 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:05.658694 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:05.658849 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:05.485915 17787 raft_consensus.cc:1226] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:05.666287 18728 raft_consensus.cc:482] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:05.667155 18903 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:05.668787 18948 consensus_queue.cc:1035] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260810 18:52:05.675271 19156 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:05.676940 19162 raft_consensus.cc:482] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:05.676025 19099 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:05.458966 19244 raft_consensus.cc:3032] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:52:05.818588 19244 raft_consensus.cc:729] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:05.820349 19244 consensus_queue.cc:260] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:05.822675 19244 raft_consensus.cc:3037] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:52:05.687649 19279 raft_consensus.cc:2781] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:05.823295 19279 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:05.823834 19279 raft_consensus.cc:3037] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:05.470278 19289 consensus_queue.cc:260] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:05.828588 19289 raft_consensus.cc:3037] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
W20260810 18:52:05.698212 19351 consensus_meta.cc:287] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 1.196s user 0.008s sys 0.016s
I20260810 18:52:05.454124 19366 raft_consensus.cc:2726] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:52:05.831190 18903 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:05.436817 19373 consensus_meta.cc:287] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 1.057s user 0.004s sys 0.010s
I20260810 18:52:05.491912 19380 raft_consensus.cc:504] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:05.834442 19758 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:05.834753 19380 leader_election.cc:288] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:05.699729 19480 raft_consensus.cc:482] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:05.835227 19099 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } }
I20260810 18:52:05.835775 19480 raft_consensus.cc:504] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:05.838182 19099 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:05.701119 19350 raft_consensus.cc:482] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:05.713425 19561 raft_consensus.cc:3037] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:05.504765 19639 raft_consensus.cc:729] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:05.503796 19658 raft_consensus.cc:729] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:05.506428 19595 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
W20260810 18:52:05.730408 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 112708us
I20260810 18:52:05.848289 19658 consensus_queue.cc:260] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:05.849532 19658 raft_consensus.cc:3037] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:05.740190 19718 raft_consensus.cc:2781] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:05.851075 19718 raft_consensus.cc:686] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:05.852586 19718 consensus_queue.cc:237] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:05.741221 19678 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:05.743595 19765 raft_consensus.cc:2781] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:05.855357 19495 raft_consensus.cc:482] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:05.855779 19765 raft_consensus.cc:686] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:05.855691 19678 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:05.856999 19099 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:05.856606 19639 consensus_queue.cc:260] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:05.857752 19639 raft_consensus.cc:3037] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:05.857508 19099 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:05.857317 19765 consensus_queue.cc:237] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:05.858688 19678 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:05.753989 19470 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:05.871249 19678 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:05.871518 19470 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:05.871910 19678 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:05.872619 16740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 160) took 1676 ms. Trace:
I20260810 18:52:05.873003 16740 rpcz_store.cc:276] 0810 18:52:04.196297 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:04.196425 (+ 128us) service_pool.cc:227] Handling call
0810 18:52:04.221658 (+ 25233us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:05.791662 (+1570004us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:05.872222 (+ 80560us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54002ee5d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:05.872450 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2642562816}
I20260810 18:52:05.873629 19470 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:05.770768 16344 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.504s user 0.002s sys 0.001s
I20260810 18:52:05.710065 19543 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:05.874272 19678 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:05.875794 19543 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:05.786583 16745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 164) took 1438 ms. Trace:
W20260810 18:52:05.811069 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 134670us
I20260810 18:52:05.810775 17786 raft_consensus.cc:2454] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:05.813689 19684 raft_consensus.cc:2781] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:52:05.810391 19156 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:05.890636 19684 raft_consensus.cc:686] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:05.891944 19684 consensus_queue.cc:237] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:05.892552 19156 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:05.815174 16501 raft_consensus.cc:1226] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 6 LEADER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:05.814023 16749 rpcz_store.cc:276] 0810 18:52:04.340470 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:04.340756 (+ 286us) service_pool.cc:227] Handling call
0810 18:52:04.341251 (+ 495us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:04.361358 (+ 20107us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:04.361410 (+ 52us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:04.395328 (+ 33918us) log.cc:842] Serialized 26 byte log entry
0810 18:52:04.842071 (+446743us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:04.849480 (+ 7409us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:04.849547 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:05.377784 (+528237us) raft_consensus.cc:1671] finished
0810 18:52:05.377838 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:05.377971 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14624,"raft.queue_time_us":15277,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":15294,"thread_start_us":359872,"threads_started":1,"wal-append.queue_time_us":902950}
I20260810 18:52:05.815287 19162 raft_consensus.cc:504] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:05.816957 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 198477us
I20260810 18:52:05.817202 16633 raft_consensus.cc:1261] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260810 18:52:05.817888 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 178261us
I20260810 18:52:05.838516 19480 leader_election.cc:288] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:05.896301 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:05.896628 19162 leader_election.cc:288] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:05.840881 17778 raft_consensus.cc:2379] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 4: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:52:05.897096 17786 raft_consensus.cc:2454] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
I20260810 18:52:05.840628 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:05.840775 19168 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:05.899523 17783 raft_consensus.cc:2454] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:05.899577 19168 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:05.900303 19838 log.cc:924] Time spent T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.109s user 0.002s sys 0.000s
I20260810 18:52:05.896322 16575 leader_election.cc:302] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:05.903465 19380 consensus_queue.cc:1046] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260810 18:52:05.905779 19814 raft_consensus.cc:2726] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:05.906613 19380 raft_consensus.cc:3032] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:52:05.813800 18728 raft_consensus.cc:504] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:05.846215 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:05.846005 19350 raft_consensus.cc:504] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:52:05.856050 19495 raft_consensus.cc:504] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:05.745561 19755 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:05.859942 19099 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:05.860543 19821 raft_consensus.cc:482] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:05.945549 19350 leader_election.cc:288] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:05.945467 19821 raft_consensus.cc:504] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:05.861207 19763 raft_consensus.cc:482] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:05.947697 19763 raft_consensus.cc:504] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:05.949199 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:05.949950 16633 raft_consensus.cc:2454] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:52:05.752853 19391 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:05.951474 19391 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:05.953106 19495 leader_election.cc:288] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:05.954289 19391 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:05.865762 19830 log.cc:924] Time spent T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.081s user 0.000s sys 0.001s
W20260810 18:52:05.866303 19827 log.cc:924] Time spent T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.085s user 0.000s sys 0.001s
I20260810 18:52:05.950620 19763 leader_election.cc:288] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:05.861078 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:05.958298 16752 raft_consensus.cc:3037] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:05.959271 16500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1361) took 1068 ms. Trace:
I20260810 18:52:05.959699 16500 rpcz_store.cc:276] 0810 18:52:04.891262 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:04.898799 (+ 7537us) service_pool.cc:227] Handling call
0810 18:52:05.002556 (+103757us) raft_consensus.cc:1457] Updating replica for 2 ops
0810 18:52:05.190339 (+187783us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:05.192732 (+ 2393us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0810 18:52:05.294161 (+101429us) log.cc:842] Serialized 52 byte log entry
0810 18:52:05.780567 (+486406us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:52:05.780611 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:05.780698 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:05.865429 (+ 84731us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b44000232cc. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae50e4a95 00007f1ae50e54b2 00007f1ae7d8d511 00007f1ae7d88e1a 00007f1ae7d7cbee 00007f1ae7d7bf36 00007f1ae7d7bb58 00007f1ae7d85551 00007f1ae7d85509 00007f1ae7d85499
0810 18:52:05.958708 (+ 93279us) raft_consensus.cc:1671] finished
0810 18:52:05.958742 (+ 34us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:05.959042 (+ 300us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":72370,"raft.queue_time_us":75904,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":16602,"spinlock_wait_cycles":35697920,"thread_start_us":235239,"threads_started":1,"wal-append.queue_time_us":477559}
I20260810 18:52:05.868884 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:05.962381 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:05.834456 18903 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:05.881585 16573 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:05.876641 16745 rpcz_store.cc:276] 0810 18:52:04.348456 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:04.348590 (+ 134us) service_pool.cc:227] Handling call
0810 18:52:04.348948 (+ 358us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:05.757000 (+1408052us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:05.786193 (+ 29193us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54002c2ed4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:05.786491 (+ 298us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2920381312}
I20260810 18:52:05.963645 16500 raft_consensus.cc:3037] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:05.883373 16695 leader_election.cc:302] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:05.963953 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:05.964632 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 81495.5us
I20260810 18:52:05.965381 18903 raft_consensus.cc:2781] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:52:05.965772 18903 raft_consensus.cc:482] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:05.966192 18903 raft_consensus.cc:3037] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:05.885367 16496 raft_consensus.cc:1261] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260810 18:52:05.886286 16495 raft_consensus.cc:1261] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260810 18:52:05.886359 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:05.887570 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:05.887712 19595 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:05.889662 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:05.906875 16575 leader_election.cc:302] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:05.962981 17780 raft_consensus.cc:2454] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:05.844298 16508 raft_consensus.cc:2454] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:05.911289 16749 raft_consensus.cc:1261] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260810 18:52:05.912592 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:05.967039 16695 leader_election.cc:302] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:05.912894 19168 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:05.913609 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:05.968729 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:05.918488 16750 raft_consensus.cc:1226] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 1. Current term is 2. Ops: []
I20260810 18:52:05.969739 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:05.920915 18728 leader_election.cc:288] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:05.923353 16746 raft_consensus.cc:1226] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:05.924273 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:05.930606 19380 raft_consensus.cc:729] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:05.967948 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:05.926223 17781 raft_consensus.cc:1261] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260810 18:52:05.928462 17786 raft_consensus.cc:1261] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260810 18:52:05.928642 17778 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55586 (request call id 296) took 1289 ms (client timeout 1616 ms). Trace:
I20260810 18:52:05.926908 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:05.864648 16701 leader_election.cc:334] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.824s (SENT)
I20260810 18:52:05.944005 19755 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:05.948370 19821 leader_election.cc:288] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:05.839771 19758 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:05.964685 16745 raft_consensus.cc:2454] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:05.973389 19814 raft_consensus.cc:2726] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:05.884559 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
W20260810 18:52:05.963888 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 82428us
W20260810 18:52:05.971825 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 65087.7us
I20260810 18:52:05.973176 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:05.973300 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:05.975468 16748 raft_consensus.cc:2454] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:05.971102 16507 raft_consensus.cc:2454] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:52:05.973690 16743 raft_consensus.cc:2454] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:52:06.011291 16749 raft_consensus.cc:2454] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:52:05.924518 16740 raft_consensus.cc:1261] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260810 18:52:05.986531 17783 raft_consensus.cc:2379] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 4: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:52:05.972640 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:05.991460 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:05.971957 19595 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:05.998307 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:06.001572 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:52:06.019279 16752 raft_consensus.cc:2454] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 4.
I20260810 18:52:05.883510 16632 raft_consensus.cc:1261] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260810 18:52:05.982884 19391 raft_consensus.cc:2726] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:52:06.020105 16622 raft_consensus.cc:1226] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260810 18:52:06.021600 19380 consensus_queue.cc:260] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:06.027806 17770 raft_consensus.cc:2454] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:06.024492 19099 raft_consensus.cc:2781] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:52:06.058673 19391 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:06.059356 19391 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:06.061514 19391 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:05.993392 17778 rpcz_store.cc:269] 0810 18:52:04.639198 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:04.646727 (+ 7529us) service_pool.cc:227] Handling call
0810 18:52:05.928292 (+1281565us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54006bd854. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:05.928540 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2738351360}
I20260810 18:52:06.029196 19029 consensus_queue.cc:1035] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260810 18:52:06.030290 16575 leader_election.cc:302] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:06.033160 16502 raft_consensus.cc:2396] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260810 18:52:06.035241 19606 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:06.008225 19763 consensus_queue.cc:1046] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.028s
I20260810 18:52:06.036068 16695 leader_election.cc:302] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:06.037302 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:06.038031 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:06.038766 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:06.036104 19815 raft_consensus.cc:482] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:06.039639 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 175052us
I20260810 18:52:06.075297 16504 raft_consensus.cc:2454] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:06.040212 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:06.045022 19758 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:06.034287 18903 raft_consensus.cc:504] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:06.079627 19815 raft_consensus.cc:504] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:06.082024 19815 leader_election.cc:288] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:06.056483 16573 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:52:06.058377 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:06.072239 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:06.077787 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 67492.7us
I20260810 18:52:06.074702 16495 raft_consensus.cc:2362] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4. Re-sending same reply.
W20260810 18:52:06.087157 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 51171.8us
I20260810 18:52:06.091308 19606 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:06.092782 16701 leader_election.cc:334] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.889s (SENT)
I20260810 18:52:06.093427 16701 leader_election.cc:302] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:06.093796 19099 raft_consensus.cc:482] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:06.094172 19755 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:06.094481 19099 raft_consensus.cc:3037] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:52:06.090342 16695 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:06.098443 19758 raft_consensus.cc:2726] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:52:06.098552 19168 consensus_queue.cc:1035] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:52:06.103582 19749 raft_consensus.cc:2781] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:06.104835 19749 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:06.105247 19749 raft_consensus.cc:3037] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:06.105262 19758 raft_consensus.cc:2781] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:06.107631 19758 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:06.107966 19758 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:06.108139 16573 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:06.111188 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:52:06.111487 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 81326.7us
I20260810 18:52:06.112121 17778 raft_consensus.cc:2379] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 5: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:52:06.111907 19839 consensus_queue.cc:1035] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:52:06.112428 16575 leader_election.cc:302] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:06.112560 19755 raft_consensus.cc:2726] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:06.113719 19366 raft_consensus.cc:2726] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:06.113935 16575 leader_election.cc:302] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:06.114236 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:06.114775 19366 raft_consensus.cc:2726] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:52:06.119282 19786 consensus_queue.cc:1035] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260810 18:52:06.096369 19815 raft_consensus.cc:2781] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:06.125981 19815 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260810 18:52:06.123862 16701 leader_election.cc:334] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.013s (SENT)
I20260810 18:52:06.102562 16506 raft_consensus.cc:2454] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
W20260810 18:52:06.127635 16701 leader_election.cc:334] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.022s (SENT)
W20260810 18:52:06.128721 16701 leader_election.cc:334] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.028s (SENT)
I20260810 18:52:06.129091 16701 leader_election.cc:302] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:06.129861 19380 raft_consensus.cc:3037] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:06.130826 16501 raft_consensus.cc:2454] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:06.126530 19815 raft_consensus.cc:3037] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:06.130717 19279 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:06.126492 16502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52338 (request call id 1362) took 1234 ms. Trace:
I20260810 18:52:06.135183 16502 rpcz_store.cc:276] 0810 18:52:04.891707 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:04.898095 (+ 6388us) service_pool.cc:227] Handling call
0810 18:52:05.554232 (+656137us) spinlock_profiling.cc:243] Waited 7.57 ms on lock 0x7b5800001898. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519abce 000000000034aea9 00007f1aecfc6a30 00007f1aed08c7b4 00007f1aed08c821 00007f1aed03c1b0 00007f1aed03ba94 00007f1ae6723c04 00007f1ae6723b92
0810 18:52:06.126217 (+571985us) spinlock_profiling.cc:243] Waited 477 ms on lock 0x7b5400116554. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:06.126402 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1114843904}
I20260810 18:52:06.134467 16500 raft_consensus.cc:2454] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:06.077872 19391 raft_consensus.cc:2726] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:52:06.139655 19279 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:06.140156 19606 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:06.141736 19099 raft_consensus.cc:504] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:06.139768 16506 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52338 (request call id 1363) took 1247 ms. Trace:
I20260810 18:52:06.145880 16506 rpcz_store.cc:276] 0810 18:52:04.892218 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:04.892383 (+ 165us) service_pool.cc:227] Handling call
0810 18:52:06.139574 (+1247191us) spinlock_profiling.cc:243] Waited 871 ms on lock 0x7b54001cb6d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:06.139705 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2004318976}
I20260810 18:52:06.147578 19763 raft_consensus.cc:3032] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:06.150390 19763 raft_consensus.cc:729] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:06.151042 19350 consensus_queue.cc:1035] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:52:06.151332 19763 consensus_queue.cc:260] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:06.151144 18903 leader_election.cc:288] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:06.152642 19763 raft_consensus.cc:3037] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:06.156011 16450 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:06.156970 19168 consensus_queue.cc:1035] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:52:06.157517 16450 leader_election.cc:302] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:06.157779 19639 raft_consensus.cc:2781] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:06.158660 19639 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:06.159004 16450 leader_election.cc:302] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:06.159142 19639 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:06.163511 19168 consensus_queue.cc:1035] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260810 18:52:06.163971 16450 leader_election.cc:302] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:06.165395 19168 raft_consensus.cc:2781] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:06.165881 19168 raft_consensus.cc:482] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:06.166347 19168 raft_consensus.cc:3037] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:06.166517 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:06.159255 19279 raft_consensus.cc:2781] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:52:06.167379 16750 raft_consensus.cc:2454] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:06.168190 16573 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:06.167455 19279 raft_consensus.cc:482] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:06.168911 19279 raft_consensus.cc:3037] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:06.169250 16573 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:06.170600 16573 leader_election.cc:302] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:06.171247 19765 raft_consensus.cc:2781] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:06.171627 19391 raft_consensus.cc:2726] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:52:06.171821 19765 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:06.171303 18903 raft_consensus.cc:2726] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:06.172287 19765 raft_consensus.cc:3037] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:06.161593 18728 raft_consensus.cc:2781] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:06.184852 18728 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:06.185559 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:06.185561 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:06.176689 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 164469us
W20260810 18:52:06.173120 19840 log.cc:924] Time spent T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.143s user 0.003s sys 0.000s
I20260810 18:52:06.195783 16573 leader_election.cc:302] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:06.201758 16450 leader_election.cc:302] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:06.186040 18728 raft_consensus.cc:3037] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:06.172406 19391 raft_consensus.cc:2781] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:06.208779 16750 raft_consensus.cc:2454] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:52:06.209683 16575 leader_election.cc:302] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:06.211577 16695 leader_election.cc:302] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:06.205452 17778 raft_consensus.cc:2454] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:06.218003 19755 raft_consensus.cc:2726] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:52:06.218185 17770 raft_consensus.cc:1261] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260810 18:52:06.219594 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:06.220199 19718 raft_consensus.cc:2781] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:52:06.221753 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:06.207231 18903 consensus_queue.cc:1046] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.016s
I20260810 18:52:06.223310 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:06.223687 18903 consensus_queue.cc:1225] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer d304ac47fdf846de81337057a4ef7bea log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260810 18:52:06.224144 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:06.224484 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:06.224817 16507 raft_consensus.cc:2454] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:06.220279 16745 raft_consensus.cc:3037] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:06.231276 16450 leader_election.cc:302] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:06.230012 19289 raft_consensus.cc:2781] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:06.232681 19289 raft_consensus.cc:482] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:06.233059 19289 raft_consensus.cc:3037] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:06.234241 19718 raft_consensus.cc:686] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 3 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:06.214002 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:06.236850 16500 raft_consensus.cc:2454] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:06.237080 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:06.240454 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:06.239988 18948 consensus_queue.cc:1035] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.004s
I20260810 18:52:06.256486 19593 raft_consensus.cc:2726] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260810 18:52:06.257157 19099 leader_election.cc:288] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:06.267742 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 81421.1us
I20260810 18:52:06.171905 19749 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:06.271306 19587 raft_consensus.cc:2781] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:52:06.271804 19587 raft_consensus.cc:482] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:06.272142 19587 raft_consensus.cc:3037] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:06.273285 16748 raft_consensus.cc:3032] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:06.273360 16632 raft_consensus.cc:1261] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260810 18:52:06.273756 16748 raft_consensus.cc:729] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:06.227943 19391 raft_consensus.cc:686] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:06.276971 16573 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:06.275837 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:06.278716 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:06.275215 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:06.274641 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:06.281775 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:06.282596 16623 raft_consensus.cc:2454] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:06.285257 16624 raft_consensus.cc:2454] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:06.285609 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:06.286223 16622 raft_consensus.cc:2454] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:06.287099 16633 raft_consensus.cc:2454] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:06.282614 16621 raft_consensus.cc:1261] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260810 18:52:06.288779 19718 consensus_queue.cc:237] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:06.291824 16575 leader_election.cc:302] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:06.293128 19755 raft_consensus.cc:2781] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:52:06.293517 19755 raft_consensus.cc:482] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:06.293951 19755 raft_consensus.cc:3037] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
W20260810 18:52:06.295847 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 84352.1us
I20260810 18:52:06.274619 16748 consensus_queue.cc:260] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:06.297881 16506 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:06.302348 19366 raft_consensus.cc:2781] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:06.302753 19366 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:06.303143 19366 raft_consensus.cc:3037] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:06.286592 19815 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:06.333710 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:06.337692 16695 leader_election.cc:302] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:06.347653 16447 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:06.351255 19684 raft_consensus.cc:2781] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:06.351719 19684 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:06.352065 19684 raft_consensus.cc:3037] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:06.318953 19156 consensus_queue.cc:1035] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.038s
I20260810 18:52:06.320981 19593 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:06.341863 19758 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:06.370100 16573 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:06.370319 16575 leader_election.cc:302] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:06.359056 17783 raft_consensus.cc:2454] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:06.359963 16448 leader_election.cc:302] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:52:06.390043 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 143958us
W20260810 18:52:06.383870 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 46270.1us
I20260810 18:52:06.390874 16745 raft_consensus.cc:2454] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:06.354453 19765 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:06.372640 19852 log.cc:924] Time spent T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.177s user 0.001s sys 0.000s
W20260810 18:52:06.359586 19845 log.cc:924] Time spent T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.250s user 0.001s sys 0.000s
I20260810 18:52:06.396402 19156 raft_consensus.cc:2781] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:06.411172 19156 raft_consensus.cc:482] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:06.411670 19156 raft_consensus.cc:3037] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:06.412006 19099 raft_consensus.cc:2781] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:52:06.412757 19099 raft_consensus.cc:686] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:06.414001 19099 consensus_queue.cc:237] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:06.368643 19815 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:06.402311 16748 raft_consensus.cc:3037] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:06.372530 18728 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:06.406301 19391 consensus_queue.cc:237] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:06.424741 18728 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:06.424950 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:06.425809 17770 raft_consensus.cc:3037] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:06.372494 19351 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:06.449853 16450 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:06.404610 17727 leader_election.cc:302] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:06.454347 19350 consensus_queue.cc:1035] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:52:06.456774 19815 raft_consensus.cc:2726] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:06.457666 19815 raft_consensus.cc:2726] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:06.360616 19593 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:06.461586 19593 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:06.463332 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:06.464147 16740 raft_consensus.cc:3037] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:06.342984 19658 consensus_queue.cc:1035] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.098s
I20260810 18:52:06.377638 19279 raft_consensus.cc:504] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:06.468973 19279 leader_election.cc:288] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:06.369422 19758 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:06.479260 19697 raft_consensus.cc:2781] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:52:06.489121 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:06.489892 16626 raft_consensus.cc:3037] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
W20260810 18:52:06.482951 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 83385.5us
W20260810 18:52:06.498523 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 268198us
I20260810 18:52:06.365065 19684 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:06.499718 16695 leader_election.cc:302] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:06.515887 16695 leader_election.cc:302] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:06.479629 19351 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:06.483852 19697 raft_consensus.cc:686] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:06.540863 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:06.550063 16507 raft_consensus.cc:1226] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260810 18:52:06.477053 19587 raft_consensus.cc:504] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:52:06.539273 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:06.560472 16747 raft_consensus.cc:3037] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:06.567080 19584 consensus_queue.cc:1035] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260810 18:52:06.571496 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:06.572422 17780 raft_consensus.cc:2454] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:06.575093 19858 raft_consensus.cc:2781] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:52:06.576999 19755 raft_consensus.cc:504] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:06.574668 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:06.581096 19755 leader_election.cc:288] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:06.587348 19749 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:06.588266 16748 raft_consensus.cc:1261] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260810 18:52:06.591338 19495 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260810 18:52:06.592525 19351 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:06.607875 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:06.608709 17786 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:06.625615 19639 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:06.516767 19593 raft_consensus.cc:2781] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:06.641096 19593 raft_consensus.cc:482] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:06.641625 19593 raft_consensus.cc:3037] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:06.644644 19684 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:06.647305 19168 raft_consensus.cc:504] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:06.649853 19168 leader_election.cc:288] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:06.497335 19289 raft_consensus.cc:504] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:06.667325 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 99573.7us
I20260810 18:52:06.669299 16748 raft_consensus.cc:1261] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:52:06.675768 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:06.676575 16624 raft_consensus.cc:3037] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:06.546164 19765 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:06.678864 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:06.683560 16748 raft_consensus.cc:3037] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 4
I20260810 18:52:06.683908 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:06.684653 16633 raft_consensus.cc:3037] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:06.558038 19595 raft_consensus.cc:2781] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:06.692003 16506 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:06.695403 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:06.695652 16740 raft_consensus.cc:2454] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:06.772512 16745 raft_consensus.cc:1226] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260810 18:52:06.777071 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:06.777925 17787 raft_consensus.cc:3037] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:06.754804 17727 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:06.783850 16506 raft_consensus.cc:2354] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3.
I20260810 18:52:06.681207 19858 raft_consensus.cc:686] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 4 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
W20260810 18:52:06.750957 19380 consensus_meta.cc:287] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.621s user 0.005s sys 0.009s
I20260810 18:52:06.694629 19595 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:06.810037 19595 raft_consensus.cc:3037] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:06.773514 19684 consensus_queue.cc:1035] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20260810 18:52:06.819370 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:06.820076 16500 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:06.820152 16636 raft_consensus.cc:1261] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260810 18:52:06.805200 19366 consensus_meta.cc:287] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.502s user 0.013s sys 0.002s
I20260810 18:52:06.788947 19162 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:06.840517 16742 raft_consensus.cc:1226] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:06.841603 19863 consensus_queue.cc:1046] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.148s
I20260810 18:52:06.842847 19863 raft_consensus.cc:3032] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:52:06.842907 19291 consensus_queue.cc:1046] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260810 18:52:06.843402 19863 raft_consensus.cc:729] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:06.844185 19291 raft_consensus.cc:3032] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:52:06.844560 19291 raft_consensus.cc:729] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:06.845530 19291 consensus_queue.cc:260] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:06.846836 19291 raft_consensus.cc:3037] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:06.811177 19697 consensus_queue.cc:237] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:06.855665 19858 consensus_queue.cc:237] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:06.859301 19366 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:06.861855 19366 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:06.872215 16748 raft_consensus.cc:2454] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:06.820077 19639 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:06.879674 16624 raft_consensus.cc:2454] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:06.875208 19162 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:06.884296 19162 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:06.886816 16747 raft_consensus.cc:2454] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:06.889827 16626 raft_consensus.cc:2454] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
W20260810 18:52:06.888341 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 518321us
I20260810 18:52:06.832736 17770 raft_consensus.cc:2454] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
W20260810 18:52:06.879626 16750 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s user 0.008s sys 0.014s
W20260810 18:52:06.895144 16750 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.600s user 0.009s sys 0.014s
W20260810 18:52:06.897600 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 527339us
I20260810 18:52:06.908399 19156 raft_consensus.cc:504] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:06.908921 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:06.910511 19156 leader_election.cc:288] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:06.910784 19289 leader_election.cc:288] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:06.933782 17787 raft_consensus.cc:2454] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:06.937525 16573 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:52:06.940074 19351 raft_consensus.cc:482] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:06.940716 19351 raft_consensus.cc:504] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:06.943216 19835 consensus_queue.cc:1035] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.112s
I20260810 18:52:06.943514 19351 leader_election.cc:288] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:06.945667 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:06.951334 17786 raft_consensus.cc:2454] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:06.955320 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:06.950903 19366 raft_consensus.cc:2781] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:52:06.959182 17778 raft_consensus.cc:2379] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 5: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:52:06.952106 19587 leader_election.cc:288] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:06.897054 19863 consensus_queue.cc:260] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:06.982460 16622 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.015s sys 0.014s
W20260810 18:52:06.993968 16622 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.534s user 0.016s sys 0.014s
W20260810 18:52:06.996333 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 67104.7us
W20260810 18:52:06.998210 16628 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.007s sys 0.025s
W20260810 18:52:06.998684 16628 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.694s user 0.007s sys 0.026s
I20260810 18:52:06.980458 19366 raft_consensus.cc:686] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 4 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:07.015774 16575 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:06.981118 19718 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.612s user 0.031s sys 0.017s
W20260810 18:52:07.017249 19718 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.649s user 0.032s sys 0.017s
I20260810 18:52:06.981841 19593 raft_consensus.cc:504] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:07.031236 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 246425us
I20260810 18:52:07.034333 16626 raft_consensus.cc:1226] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 1. Current term is 2. Ops: []
W20260810 18:52:07.030728 16631 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.614s user 0.002s sys 0.070s
W20260810 18:52:07.038815 16631 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.675s user 0.002s sys 0.070s
I20260810 18:52:07.040207 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:07.040846 16626 raft_consensus.cc:3037] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:07.043423 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:07.044153 16632 raft_consensus.cc:3037] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:07.026587 19718 raft_consensus.cc:2726] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:07.057176 16633 raft_consensus.cc:2454] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
W20260810 18:52:07.056613 16695 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.007s sys 0.015s
W20260810 18:52:07.059278 16695 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.505s user 0.007s sys 0.015s
W20260810 18:52:07.059755 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 505198us
I20260810 18:52:07.040251 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:07.061372 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:07.069482 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:07.068444 16695 leader_election.cc:302] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:07.072126 16695 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
W20260810 18:52:07.074465 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 61288.4us
I20260810 18:52:07.075371 16624 raft_consensus.cc:2454] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:07.080605 16626 raft_consensus.cc:2454] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:07.080045 16695 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:07.073417 19099 raft_consensus.cc:2781] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:52:07.086321 16501 raft_consensus.cc:1226] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:07.085991 17778 raft_consensus.cc:2354] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for earlier term 4. Current term is 5.
I20260810 18:52:07.091354 16575 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:07.081089 19758 raft_consensus.cc:2781] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:07.093467 17786 raft_consensus.cc:1226] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260810 18:52:07.103386 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:07.103963 17787 raft_consensus.cc:3037] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:07.090624 19156 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:07.109592 19156 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:07.110069 16448 leader_election.cc:302] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:07.111282 16448 leader_election.cc:302] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:07.112151 19156 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:07.112546 19561 raft_consensus.cc:482] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:07.113245 19168 raft_consensus.cc:2781] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:52:07.113812 19168 raft_consensus.cc:686] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:07.113328 19561 raft_consensus.cc:504] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:07.114743 19168 consensus_queue.cc:237] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:07.115763 19289 raft_consensus.cc:2781] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:52:07.116443 19289 raft_consensus.cc:686] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:07.117643 19289 consensus_queue.cc:237] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:07.112764 19912 raft_consensus.cc:482] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:07.120389 19912 raft_consensus.cc:504] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:07.123868 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:07.124473 16495 raft_consensus.cc:3037] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:07.125705 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:07.073535 19366 consensus_queue.cc:237] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:07.089139 19495 raft_consensus.cc:2781] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:07.133793 16495 raft_consensus.cc:2454] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
W20260810 18:52:07.138422 17726 leader_election.cc:334] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.663s (SENT)
I20260810 18:52:07.138839 19561 leader_election.cc:288] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:07.136613 19029 raft_consensus.cc:2726] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:07.094164 19099 raft_consensus.cc:686] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:07.141088 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:07.100627 19758 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:07.144042 17787 raft_consensus.cc:2454] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:07.099788 19891 raft_consensus.cc:2726] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:07.138963 17726 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:07.112756 19156 consensus_queue.cc:1035] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.027s
W20260810 18:52:07.149240 19895 log.cc:924] Time spent T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20260810 18:52:07.151221 16695 leader_election.cc:302] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:07.154999 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:07.157823 16448 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:07.161892 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 74352.2us
I20260810 18:52:07.141629 19863 raft_consensus.cc:3037] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:07.168995 16448 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:07.168615 19162 raft_consensus.cc:2781] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:07.175570 19162 raft_consensus.cc:686] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:07.177312 19639 raft_consensus.cc:2781] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:07.177757 19639 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:07.178125 19639 raft_consensus.cc:3037] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:07.165205 19858 raft_consensus.cc:2726] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:52:07.179441 19593 leader_election.cc:288] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:07.180243 19593 raft_consensus.cc:2781] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:07.180707 19593 raft_consensus.cc:686] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:07.181644 19593 consensus_queue.cc:237] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:07.181644 16448 leader_election.cc:302] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:07.183264 19162 consensus_queue.cc:237] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:07.194092 16495 raft_consensus.cc:2454] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:07.194690 19561 raft_consensus.cc:2781] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:52:07.202010 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:07.151607 19758 raft_consensus.cc:3037] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
W20260810 18:52:07.217092 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 78846.8us
I20260810 18:52:07.220470 16447 leader_election.cc:398] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Vote denied by peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387) with higher term. Message: Invalid argument: T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for earlier term 4. Current term is 5.
I20260810 18:52:07.220901 16447 leader_election.cc:401] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260810 18:52:07.222265 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:07.223186 16622 raft_consensus.cc:2454] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:07.224637 16632 raft_consensus.cc:2454] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:07.150444 19350 consensus_queue.cc:1046] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.066s
I20260810 18:52:07.237641 16575 leader_election.cc:302] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:52:07.235131 17726 leader_election.cc:334] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.708s (SENT)
I20260810 18:52:07.238780 17726 leader_election.cc:302] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:07.245427 16573 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:07.247674 19899 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:07.248383 19899 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:07.249001 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:07.249815 17787 raft_consensus.cc:2454] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:07.271736 16447 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:07.273480 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:07.202499 19391 consensus_queue.cc:1035] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.075s
I20260810 18:52:07.251019 19351 raft_consensus.cc:2781] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election won for term 5
W20260810 18:52:07.280807 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 487839us
I20260810 18:52:07.288244 16450 leader_election.cc:302] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:07.207084 19891 consensus_queue.cc:1035] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.067s
I20260810 18:52:07.254194 19858 raft_consensus.cc:2726] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260810 18:52:07.250977 19899 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:07.209724 16745 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b"
dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
mode: GRACEFUL
new_leader_uuid: "d304ac47fdf846de81337057a4ef7bea"
from {username='slave'} at 127.0.0.1:45870
I20260810 18:52:07.297175 16745 raft_consensus.cc:595] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Received request to transfer leadership to TS d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:07.298084 16506 raft_consensus.cc:1261] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260810 18:52:07.307657 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:07.309702 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:07.310546 16740 raft_consensus.cc:2454] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:07.294685 19718 raft_consensus.cc:2726] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260810 18:52:07.281214 19099 consensus_queue.cc:237] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:07.295166 19351 raft_consensus.cc:686] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:07.321240 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:07.321911 16750 raft_consensus.cc:3037] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
W20260810 18:52:07.328826 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 100675us
I20260810 18:52:07.307482 19697 raft_consensus.cc:2781] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:52:07.318595 19899 raft_consensus.cc:2781] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:07.347353 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:07.348109 16747 raft_consensus.cc:3037] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
W20260810 18:52:07.331450 19595 consensus_meta.cc:287] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.515s user 0.008s sys 0.003s
I20260810 18:52:07.310420 19912 leader_election.cc:288] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:07.338605 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
W20260810 18:52:07.360643 16701 leader_election.cc:334] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.508s (SENT)
I20260810 18:52:07.315964 19561 raft_consensus.cc:686] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
W20260810 18:52:07.349725 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 201568us
I20260810 18:52:07.361263 16701 leader_election.cc:302] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:07.359427 16500 consensus_meta.cc:287] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.539s user 0.004s sys 0.008s
I20260810 18:52:07.341967 19495 raft_consensus.cc:686] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:07.356756 19899 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260810 18:52:07.383311 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 187510us
I20260810 18:52:07.364761 19289 raft_consensus.cc:482] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:07.403548 16695 leader_election.cc:302] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:07.359040 19595 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:07.398734 19899 raft_consensus.cc:3037] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:07.383610 16500 raft_consensus.cc:2454] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
W20260810 18:52:07.419028 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 159581us
I20260810 18:52:07.420764 16448 leader_election.cc:302] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:07.444871 16573 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
W20260810 18:52:07.448285 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 236980us
I20260810 18:52:07.433380 16747 raft_consensus.cc:2454] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:07.448748 16573 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:07.411195 19763 raft_consensus.cc:2726] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260810 18:52:07.421248 19900 log.cc:924] Time spent T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.253s user 0.000s sys 0.001s
I20260810 18:52:07.397692 19639 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:07.418290 19907 raft_consensus.cc:3032] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:52:07.453505 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:07.457453 16575 leader_election.cc:302] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:07.406520 19896 consensus_queue.cc:1046] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20260810 18:52:07.461388 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:07.465970 16740 raft_consensus.cc:2454] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:07.438704 19350 raft_consensus.cc:2726] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387) with higher term. Message: Invalid argument: T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for earlier term 4. Current term is 5.
I20260810 18:52:07.468537 16622 raft_consensus.cc:2454] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
W20260810 18:52:07.469026 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 112873us
I20260810 18:52:07.411984 19289 raft_consensus.cc:504] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:07.479480 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:07.486377 16701 leader_election.cc:334] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.637s (SENT)
I20260810 18:52:07.486963 16701 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:07.460156 19907 raft_consensus.cc:729] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
W20260810 18:52:07.510664 19919 log.cc:924] Time spent T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.173s user 0.000s sys 0.001s
W20260810 18:52:07.512190 19911 log.cc:924] Time spent T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.199s user 0.001s sys 0.001s
I20260810 18:52:07.488466 19561 consensus_queue.cc:237] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:07.488353 19495 consensus_queue.cc:237] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:07.412257 19351 consensus_queue.cc:237] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:07.521349 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 51687us
I20260810 18:52:07.509423 19935 raft_consensus.cc:2781] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:52:07.510123 19862 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:07.511760 19940 raft_consensus.cc:2726] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260810 18:52:07.532805 19922 log.cc:924] Time spent T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.081s user 0.000s sys 0.001s
I20260810 18:52:07.533527 19595 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:07.533869 19917 log.cc:924] Time spent T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.216s user 0.000s sys 0.001s
I20260810 18:52:07.534020 19289 leader_election.cc:288] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:07.534080 19907 consensus_queue.cc:260] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:07.534845 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 48525.1us
I20260810 18:52:07.535029 19639 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:07.572798 19923 log.cc:924] Time spent T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20260810 18:52:07.535764 19937 raft_consensus.cc:2726] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:07.539367 17783 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55570 (request call id 407) took 1023 ms. Trace:
I20260810 18:52:07.544651 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:07.565234 16448 leader_election.cc:302] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:07.471160 16631 raft_consensus.cc:2932] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 2 to 4, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:52:07.566567 19944 raft_consensus.cc:482] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:07.568816 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:07.529148 19765 raft_consensus.cc:2781] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election won for term 5
W20260810 18:52:07.582187 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 133533us
I20260810 18:52:07.599661 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:07.475107 19697 raft_consensus.cc:686] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:07.539994 19926 raft_consensus.cc:2781] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election won for term 6
W20260810 18:52:07.602957 16701 leader_election.cc:334] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.662s (SENT)
I20260810 18:52:07.602718 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:07.616043 16633 raft_consensus.cc:3037] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 5
I20260810 18:52:07.615918 16701 leader_election.cc:302] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:07.591892 19935 raft_consensus.cc:686] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:07.604882 17783 rpcz_store.cc:276] 0810 18:52:06.515899 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:06.535362 (+ 19463us) service_pool.cc:227] Handling call
0810 18:52:06.535751 (+ 389us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:06.667291 (+131540us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:06.667333 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:06.667894 (+ 561us) log.cc:842] Serialized 26 byte log entry
0810 18:52:07.025966 (+358072us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:07.026001 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:07.026070 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:07.539132 (+513062us) raft_consensus.cc:1671] finished
0810 18:52:07.539154 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:07.539281 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17105,"raft.queue_time_us":126942,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":120,"thread_start_us":248815,"threads_started":1,"wal-append.queue_time_us":412573}
I20260810 18:52:07.606998 16745 raft_consensus.cc:2454] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:07.629339 16624 raft_consensus.cc:2454] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:07.605381 19595 raft_consensus.cc:2726] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260810 18:52:07.632314 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 174963us
W20260810 18:52:07.632983 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 67858.6us
W20260810 18:52:07.638088 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 254672us
W20260810 18:52:07.635839 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 115322us
I20260810 18:52:07.645044 16695 leader_election.cc:398] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Vote denied by peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791) with higher term. Message: Invalid argument: T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3.
I20260810 18:52:07.650527 16695 leader_election.cc:401] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260810 18:52:07.576072 19862 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:07.662380 16508 raft_consensus.cc:2454] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:07.636255 19584 raft_consensus.cc:2726] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:52:07.640111 19162 raft_consensus.cc:2781] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260810 18:52:07.665653 17726 leader_election.cc:334] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.913s (SENT)
I20260810 18:52:07.668712 16448 leader_election.cc:302] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:07.665081 16387 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.14.32.254:46351
I20260810 18:52:07.636364 19765 raft_consensus.cc:686] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:07.643695 19926 raft_consensus.cc:686] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
W20260810 18:52:07.680507 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 82858us
W20260810 18:52:07.683225 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 160315us
I20260810 18:52:07.656476 16621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1521) took 1184 ms. Trace:
I20260810 18:52:07.685184 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:07.687744 16746 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7ef7c65533724e678b75830002906e2c"
dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
mode: GRACEFUL
new_leader_uuid: "d304ac47fdf846de81337057a4ef7bea"
from {username='slave'} at 127.0.0.1:45870
I20260810 18:52:07.696607 17783 raft_consensus.cc:3037] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:07.646034 19697 consensus_queue.cc:237] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:07.688305 19765 consensus_queue.cc:237] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:07.646589 19888 consensus_queue.cc:1035] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.043s
I20260810 18:52:07.690212 19926 consensus_queue.cc:237] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:07.690868 19835 raft_consensus.cc:2781] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260810 18:52:07.694212 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 94689.7us
I20260810 18:52:07.696739 16746 raft_consensus.cc:595] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Received request to transfer leadership to TS d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:07.688935 17778 raft_consensus.cc:1261] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260810 18:52:07.700690 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:07.614539 19944 raft_consensus.cc:504] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:07.677597 19162 raft_consensus.cc:482] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:07.698947 16633 raft_consensus.cc:2454] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
W20260810 18:52:07.707702 19925 log.cc:924] Time spent T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260810 18:52:07.676220 16750 raft_consensus.cc:2454] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
W20260810 18:52:07.712816 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 171592us
I20260810 18:52:07.714680 17783 raft_consensus.cc:2454] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:07.683463 19584 raft_consensus.cc:2726] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791) with higher term. Message: Invalid argument: T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3.
I20260810 18:52:07.696354 16621 rpcz_store.cc:276] 0810 18:52:06.472243 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:06.484158 (+ 11915us) service_pool.cc:227] Handling call
0810 18:52:06.559089 (+ 74931us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:06.820189 (+261100us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:06.820233 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:06.820719 (+ 486us) log.cc:842] Serialized 26 byte log entry
0810 18:52:07.036596 (+215877us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:07.060048 (+ 23452us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:07.071472 (+ 11424us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:07.634344 (+562872us) raft_consensus.cc:1671] finished
0810 18:52:07.636406 (+ 2062us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:07.646882 (+ 10476us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":77117,"raft.queue_time_us":95185,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":22407,"thread_start_us":206765,"threads_started":1,"wal-append.queue_time_us":385409}
I20260810 18:52:07.711194 19593 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:07.721163 19593 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:07.723477 19945 raft_consensus.cc:482] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:07.727586 16695 leader_election.cc:302] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:07.713475 19835 raft_consensus.cc:482] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:07.717787 19162 raft_consensus.cc:3037] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:07.741099 16448 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:07.749580 16701 leader_election.cc:334] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.517s (SENT)
I20260810 18:52:07.750222 16701 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:07.742830 19835 raft_consensus.cc:3037] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:07.761029 19963 raft_consensus.cc:2781] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:52:07.761708 19963 raft_consensus.cc:686] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:07.762755 19963 consensus_queue.cc:237] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:07.764519 19593 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:07.766068 19953 log.cc:924] Time spent T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
W20260810 18:52:07.766299 19758 consensus_meta.cc:287] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.544s user 0.004s sys 0.006s
I20260810 18:52:07.768575 16746 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb"
dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
mode: GRACEFUL
new_leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
from {username='slave'} at 127.0.0.1:45870
I20260810 18:52:07.769389 16746 raft_consensus.cc:595] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Received request to transfer leadership to TS 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:07.769815 16746 raft_consensus.cc:603] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260810 18:52:07.728453 19945 raft_consensus.cc:504] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:07.754230 19099 raft_consensus.cc:2726] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:52:07.745321 19029 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:07.785562 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 73280us
I20260810 18:52:07.787024 19972 raft_consensus.cc:2781] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:07.787916 19029 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:07.788954 16746 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3"
dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
mode: GRACEFUL
new_leader_uuid: "d304ac47fdf846de81337057a4ef7bea"
from {username='slave'} at 127.0.0.1:45870
I20260810 18:52:07.789585 16746 raft_consensus.cc:595] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Received request to transfer leadership to TS d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:07.789938 16746 raft_consensus.cc:603] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260810 18:52:07.790098 19029 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:07.792510 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:07.792743 19945 leader_election.cc:288] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:07.803020 16450 leader_election.cc:302] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:07.806036 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:07.808056 16746 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712"
dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
mode: GRACEFUL
new_leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
from {username='slave'} at 127.0.0.1:45870
I20260810 18:52:07.808244 16632 raft_consensus.cc:2454] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 1.
I20260810 18:52:07.809655 19944 leader_election.cc:288] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:07.769002 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:07.817480 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:07.818290 16628 raft_consensus.cc:2354] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 3. Current term is 4.
I20260810 18:52:07.818629 16623 raft_consensus.cc:1203] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[3.2-5.3] Dedup: 5.3->[]
I20260810 18:52:07.819267 16623 raft_consensus.cc:1226] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 5. Current term is 6. Ops: [3.2-5.3]
I20260810 18:52:07.819434 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:07.819837 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:07.820204 19972 raft_consensus.cc:686] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:07.820470 16622 raft_consensus.cc:3037] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:07.820384 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:07.820823 16626 raft_consensus.cc:1226] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260810 18:52:07.821074 16623 raft_consensus.cc:3037] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:52:07.823669 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:07.824311 16636 raft_consensus.cc:3037] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:52:07.819335 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:07.798282 19391 raft_consensus.cc:482] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:07.819340 19913 log.cc:924] Time spent T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.471s user 0.003s sys 0.001s
I20260810 18:52:07.827395 19391 raft_consensus.cc:504] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:07.829654 19391 leader_election.cc:288] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:07.830437 17781 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55570 (request call id 408) took 1314 ms. Trace:
I20260810 18:52:07.766872 19758 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:07.832733 19758 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:07.833988 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:07.835435 17778 raft_consensus.cc:3037] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:07.806859 19905 raft_consensus.cc:2932] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Committing config change with OpId 3.4: config changed from index 2 to 4, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
W20260810 18:52:07.838476 17726 leader_election.cc:334] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.856s (SENT)
I20260810 18:52:07.838959 17726 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:07.807566 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:07.830863 17781 rpcz_store.cc:276] 0810 18:52:06.516249 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:06.539719 (+ 23470us) service_pool.cc:227] Handling call
0810 18:52:06.566363 (+ 26644us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:06.682356 (+115993us) raft_consensus.cc:1520] Early marking committed up to index 3
0810 18:52:06.687000 (+ 4644us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:06.823986 (+136986us) log.cc:842] Serialized 26 byte log entry
0810 18:52:07.098502 (+274516us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:07.105874 (+ 7372us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:07.135180 (+ 29306us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:07.818739 (+683559us) spinlock_profiling.cc:243] Waited 4.21 ms on lock 0x7b44000232cc. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae50e4a95 00007f1ae50e54b2 00007f1ae7d8d511 00007f1ae7d88e1a 00007f1ae7d7cbee 00007f1ae7d7bf36 00007f1ae7d7bb58 00007f1ae7d85551 00007f1ae7d85509 00007f1ae7d85499
0810 18:52:07.830123 (+ 11384us) raft_consensus.cc:1671] finished
0810 18:52:07.830150 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:07.830351 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":97778,"raft.queue_time_us":101119,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":9695,"spinlock_wait_cycles":9673088,"thread_start_us":265324,"threads_started":1,"wal-append.queue_time_us":471219}
I20260810 18:52:07.840301 19029 raft_consensus.cc:2726] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260810 18:52:07.709884 19935 consensus_queue.cc:237] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:07.840883 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:07.841814 16621 raft_consensus.cc:2379] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 2: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:07.834734 19929 raft_consensus.cc:3032] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:52:07.832252 16623 raft_consensus.cc:2454] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
W20260810 18:52:07.843461 17726 leader_election.cc:334] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.990s (SENT)
I20260810 18:52:07.844054 17726 leader_election.cc:302] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:07.808656 16746 raft_consensus.cc:595] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Received request to transfer leadership to TS 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:07.845309 16746 raft_consensus.cc:603] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260810 18:52:07.846829 19029 raft_consensus.cc:2726] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:07.847518 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260810 18:52:07.849078 16624 raft_consensus.cc:1226] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 LEADER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:07.849251 16746 raft_consensus.cc:1261] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260810 18:52:07.851218 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:07.852068 16624 raft_consensus.cc:2354] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 4. Current term is 5.
W20260810 18:52:07.852648 19933 log.cc:924] Time spent T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.320s user 0.000s sys 0.001s
I20260810 18:52:07.852947 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:07.853613 16621 raft_consensus.cc:3037] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:07.880578 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:07.897329 17726 leader_election.cc:334] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.782s (SENT)
I20260810 18:52:07.856341 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:07.880947 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:07.906806 16628 raft_consensus.cc:3037] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:07.907342 16624 raft_consensus.cc:3037] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
W20260810 18:52:07.871508 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 73560.8us
I20260810 18:52:07.867292 16622 raft_consensus.cc:2454] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:07.883980 16636 raft_consensus.cc:2454] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
W20260810 18:52:07.871891 16701 leader_election.cc:334] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.618s (SENT)
I20260810 18:52:07.893019 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:07.911944 16626 raft_consensus.cc:2454] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:07.855587 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:07.878046 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 75159us
I20260810 18:52:07.913147 16632 raft_consensus.cc:2454] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
W20260810 18:52:07.869940 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 123782us
I20260810 18:52:07.876596 17778 raft_consensus.cc:2454] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:07.914512 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:07.915220 16746 raft_consensus.cc:2454] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 3.
I20260810 18:52:07.915570 19835 raft_consensus.cc:504] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:07.915771 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:07.880586 19944 raft_consensus.cc:2781] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:07.917243 19944 raft_consensus.cc:482] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:07.918325 16695 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:07.919240 16450 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:07.919508 17727 leader_election.cc:302] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:07.917668 19944 raft_consensus.cc:3037] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:07.921129 16448 leader_election.cc:302] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:07.846771 19972 consensus_queue.cc:237] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:07.925055 19996 raft_consensus.cc:2726] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:07.925485 17781 raft_consensus.cc:1261] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260810 18:52:07.926187 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:07.926966 16636 raft_consensus.cc:2354] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3.
I20260810 18:52:07.842842 19929 raft_consensus.cc:729] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:07.927359 19940 consensus_queue.cc:1035] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
W20260810 18:52:07.934034 16701 leader_election.cc:334] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.693s (SENT)
I20260810 18:52:07.934619 16701 leader_election.cc:302] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:07.936260 16624 raft_consensus.cc:2454] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:07.941687 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:07.942619 16450 leader_election.cc:302] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:07.917603 19835 leader_election.cc:288] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:07.872051 19862 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:07.948307 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:07.953794 19495 raft_consensus.cc:2726] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260810 18:52:07.954703 19961 raft_consensus.cc:2781] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:07.955268 19961 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:07.955701 19961 raft_consensus.cc:3037] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:07.958509 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:07.871935 19029 consensus_queue.cc:1035] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.070s
I20260810 18:52:07.932551 19758 raft_consensus.cc:2781] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:07.964165 16695 leader_election.cc:302] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:07.968237 16506 raft_consensus.cc:2454] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 2.
I20260810 18:52:07.802565 19899 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:07.934461 19697 consensus_queue.cc:1035] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20260810 18:52:07.976298 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:07.977253 16636 raft_consensus.cc:1261] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260810 18:52:07.978148 16621 raft_consensus.cc:2454] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
W20260810 18:52:07.979274 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 45309us
I20260810 18:52:07.979918 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:07.981595 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:07.982280 19982 raft_consensus.cc:2781] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:52:07.982913 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:07.928390 19929 consensus_queue.cc:260] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:07.983736 17778 raft_consensus.cc:1261] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260810 18:52:07.990505 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:07.991364 16505 raft_consensus.cc:1261] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260810 18:52:07.989143 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:07.991969 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:07.997229 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:07.999830 16747 raft_consensus.cc:1261] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260810 18:52:08.002166 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:07.984915 16632 raft_consensus.cc:1261] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260810 18:52:08.011381 16742 raft_consensus.cc:3037] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:08.013659 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:08.006697 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:08.016134 16745 raft_consensus.cc:1261] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260810 18:52:08.007998 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 43928.6us
I20260810 18:52:08.018700 16622 raft_consensus.cc:2379] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 3: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:08.064872 16750 raft_consensus.cc:3037] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:07.968426 19996 raft_consensus.cc:2781] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:52:08.287690 16752 raft_consensus.cc:2454] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:52:08.102907 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:08.289915 16633 raft_consensus.cc:2379] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 4: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:08.055580 16628 raft_consensus.cc:2454] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:08.074892 16631 raft_consensus.cc:1261] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260810 18:52:08.296633 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:08.219368 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 44068.5us
I20260810 18:52:08.299918 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:08.188628 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:52:08.283124 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 270118us
I20260810 18:52:08.312019 16636 raft_consensus.cc:3037] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:08.313030 16743 raft_consensus.cc:2454] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:08.296100 16624 raft_consensus.cc:2454] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:08.314085 16507 raft_consensus.cc:3037] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
W20260810 18:52:08.317623 16701 leader_election.cc:334] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.744s (SENT)
I20260810 18:52:08.283236 16749 raft_consensus.cc:2932] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 2 to 4, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:52:08.318260 16701 leader_election.cc:302] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:08.018810 19099 raft_consensus.cc:2781] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:52:08.285288 19982 raft_consensus.cc:482] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:08.184340 16502 raft_consensus.cc:1261] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260810 18:52:08.238588 19936 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:08.358772 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:08.019541 19758 raft_consensus.cc:686] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:08.349704 16501 raft_consensus.cc:1226] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 3. Current term is 4. Ops: []
W20260810 18:52:08.213809 19977 log.cc:924] Time spent T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.291s user 0.000s sys 0.001s
I20260810 18:52:08.362576 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:08.374411 19162 consensus_meta.cc:287] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.617s user 0.008s sys 0.000s
I20260810 18:52:08.228472 19948 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:08.276242 19391 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:08.302294 19929 raft_consensus.cc:3037] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:08.174120 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:08.466882 16750 raft_consensus.cc:2454] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:08.327975 19996 raft_consensus.cc:686] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:08.495891 19758 consensus_queue.cc:237] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:08.361003 19982 raft_consensus.cc:3037] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:08.318279 19944 raft_consensus.cc:504] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:08.539053 19934 raft_consensus.cc:482] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:08.380514 19168 consensus_queue.cc:1035] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.169s
I20260810 18:52:08.536114 19899 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:08.509214 19391 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:08.449858 19162 raft_consensus.cc:504] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:08.556471 19244 consensus_queue.cc:1035] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.018s
W20260810 18:52:08.578449 20008 log.cc:924] Time spent T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.212s user 0.004s sys 0.000s
I20260810 18:52:08.421610 19561 consensus_queue.cc:1035] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.116s
W20260810 18:52:08.602963 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 257990us
I20260810 18:52:08.523630 19936 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:08.613243 19606 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:08.639411 19905 consensus_meta.cc:287] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.802s user 0.010s sys 0.004s
I20260810 18:52:08.467353 19948 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:08.690109 16636 raft_consensus.cc:2454] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:08.696823 16742 raft_consensus.cc:2454] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:08.705883 19944 leader_election.cc:288] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:08.899227 16749 consensus_meta.cc:287] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.534s user 0.013s sys 0.000s
I20260810 18:52:09.114217 16636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 1562) took 1010 ms. Trace:
W20260810 18:52:08.982210 16622 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.564s user 0.006s sys 0.021s
W20260810 18:52:09.137647 16622 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.719s user 0.010s sys 0.022s
I20260810 18:52:09.114596 16636 rpcz_store.cc:276] 0810 18:52:08.103785 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:08.136507 (+ 32722us) service_pool.cc:227] Handling call
0810 18:52:09.114161 (+977654us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 18:52:09.109316 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 791663us
W20260810 18:52:09.141211 16701 leader_election.cc:334] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.757s (SENT)
I20260810 18:52:09.141806 16701 leader_election.cc:302] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:08.894989 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 162304us
I20260810 18:52:09.144019 16742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43738 (request call id 1585) took 1153 ms. Trace:
I20260810 18:52:09.139861 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:09.144081 16450 leader_election.cc:302] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:09.146669 16742 rpcz_store.cc:276] 0810 18:52:07.960360 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:07.960505 (+ 145us) service_pool.cc:227] Handling call
0810 18:52:09.113273 (+1152768us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:09.189020 17778 raft_consensus.cc:3037] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:09.109915 19907 raft_consensus.cc:3037] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 2
W20260810 18:52:09.103612 19961 consensus_meta.cc:287] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 1.147s user 0.011s sys 0.004s
W20260810 18:52:09.106645 19929 consensus_meta.cc:287] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.628s user 0.011s sys 0.000s
W20260810 18:52:09.200424 17787 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.187s user 0.005s sys 0.012s
W20260810 18:52:09.200904 17787 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.188s user 0.005s sys 0.012s
I20260810 18:52:09.004120 19391 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:08.742130 19162 leader_election.cc:288] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:09.139565 19786 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:08.711691 19684 consensus_queue.cc:1035] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.487s
I20260810 18:52:09.144996 19099 raft_consensus.cc:686] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
W20260810 18:52:09.206259 16573 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.820s user 0.041s sys 0.025s
W20260810 18:52:09.206427 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 62447.7us
I20260810 18:52:09.106178 19982 raft_consensus.cc:504] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:09.241695 16573 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.855s user 0.041s sys 0.027s
W20260810 18:52:09.243750 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 857251us
I20260810 18:52:09.144845 19606 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:09.245740 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:09.243043 16450 leader_election.cc:302] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:09.247671 17770 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.868s user 0.003s sys 0.015s
W20260810 18:52:09.248070 17770 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.868s user 0.003s sys 0.015s
I20260810 18:52:09.248174 19944 raft_consensus.cc:2781] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:09.248728 19944 raft_consensus.cc:686] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:09.249245 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:09.249665 19944 consensus_queue.cc:237] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:09.248731 19156 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.736s user 0.003s sys 0.007s
W20260810 18:52:09.254772 19156 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.748s user 0.003s sys 0.007s
I20260810 18:52:09.246876 19606 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:09.260573 16748 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.580s user 0.004s sys 0.012s
W20260810 18:52:09.261976 16748 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.582s user 0.005s sys 0.013s
I20260810 18:52:09.263711 16748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1595) took 1271 ms. Trace:
W20260810 18:52:09.271868 16507 consensus_meta.cc:287] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.920s user 0.010s sys 0.001s
I20260810 18:52:09.272357 16507 raft_consensus.cc:2454] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:08.685984 19996 consensus_queue.cc:237] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:52:09.272979 16507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52352 (request call id 1438) took 1283 ms. Trace:
I20260810 18:52:09.264571 16748 rpcz_store.cc:276] 0810 18:52:07.992564 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:08.024417 (+ 31853us) service_pool.cc:227] Handling call
0810 18:52:08.276009 (+251592us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:08.276255 (+ 246us) raft_consensus.cc:1520] Early marking committed up to index 3
0810 18:52:08.680180 (+403925us) log.cc:842] Serialized 14 byte log entry
0810 18:52:09.262703 (+582523us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:09.262787 (+ 84us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:09.262870 (+ 83us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:09.263055 (+ 185us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:09.263418 (+ 363us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":970546,"threads_started":2}
I20260810 18:52:09.273379 16507 rpcz_store.cc:276] 0810 18:52:07.989987 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:07.990157 (+ 170us) service_pool.cc:227] Handling call
0810 18:52:09.272897 (+1282740us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:08.707046 19934 raft_consensus.cc:504] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:09.244182 19929 raft_consensus.cc:2726] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260810 18:52:09.289535 19987 raft_consensus.cc:2781] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260810 18:52:09.307325 16448 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.957s user 0.007s sys 0.008s
W20260810 18:52:09.307690 16448 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.979s user 0.007s sys 0.008s
W20260810 18:52:09.307986 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 999727us
W20260810 18:52:09.304067 19912 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.836s user 0.007s sys 0.007s
I20260810 18:52:09.202212 19961 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:09.313776 16695 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.002s user 0.002s sys 0.014s
I20260810 18:52:09.295804 19943 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:09.318861 19914 consensus_queue.cc:1035] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:52:09.321171 20023 consensus_queue.cc:1035] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260810 18:52:09.309458 19987 raft_consensus.cc:482] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260810 18:52:09.325875 16628 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.012s sys 0.008s
W20260810 18:52:09.337126 16628 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.627s user 0.012s sys 0.008s
W20260810 18:52:09.330153 16695 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.019s user 0.002s sys 0.014s
I20260810 18:52:09.324671 16500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1397) took 1353 ms. Trace:
W20260810 18:52:09.314253 19912 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.856s user 0.008s sys 0.008s
I20260810 18:52:09.328277 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:09.338116 16628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1564) took 1181 ms. Trace:
W20260810 18:52:09.342991 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 197056us
I20260810 18:52:09.344655 16742 raft_consensus.cc:2454] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 4.
I20260810 18:52:09.345876 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:09.343333 16500 rpcz_store.cc:276] 0810 18:52:07.971330 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:07.976357 (+ 5027us) service_pool.cc:227] Handling call
0810 18:52:07.976724 (+ 367us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:07.988318 (+ 11594us) raft_consensus.cc:1520] Early marking committed up to index 3
0810 18:52:08.236211 (+247893us) log.cc:842] Serialized 14 byte log entry
0810 18:52:08.913934 (+677723us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:09.109948 (+196014us) log.cc:842] Serialized 26 byte log entry
0810 18:52:09.110089 (+ 141us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:09.110122 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:09.110192 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:09.324384 (+214192us) raft_consensus.cc:1671] finished
0810 18:52:09.324413 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:09.324593 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":172059,"prepare.queue_time_us":1316191,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":185631,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":184,"thread_start_us":496303,"threads_started":2,"wal-append.queue_time_us":1033931}
I20260810 18:52:09.275442 19786 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:09.333446 19929 raft_consensus.cc:2726] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:52:09.330727 20024 raft_consensus.cc:482] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:09.344553 16628 rpcz_store.cc:276] 0810 18:52:08.156499 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:08.314538 (+158039us) service_pool.cc:227] Handling call
0810 18:52:08.314832 (+ 294us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:08.315031 (+ 199us) raft_consensus.cc:1520] Early marking committed up to index 6
0810 18:52:08.710063 (+395032us) log.cc:842] Serialized 14 byte log entry
0810 18:52:09.337547 (+627484us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:09.337597 (+ 50us) raft_consensus.cc:1631] Marking committed up to 6
0810 18:52:09.337871 (+ 274us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:09.337938 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:09.338060 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":763781,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"thread_start_us":918854,"threads_started":2}
I20260810 18:52:09.342435 19987 raft_consensus.cc:3037] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:09.363644 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:09.365762 20024 raft_consensus.cc:504] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:09.357307 20049 raft_consensus.cc:482] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:09.369725 20049 raft_consensus.cc:504] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:09.421698 19765 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:09.422499 19765 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:09.426865 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 106558us
I20260810 18:52:09.357143 20037 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:09.362190 19929 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:09.429844 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 138379us
I20260810 18:52:09.497551 16746 raft_consensus.cc:3032] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:52:09.497997 16746 raft_consensus.cc:729] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
W20260810 18:52:09.498049 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 68350.6us
I20260810 18:52:09.498878 16746 consensus_queue.cc:260] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:09.499991 16746 raft_consensus.cc:3037] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:09.501103 16622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1563) took 1345 ms. Trace:
I20260810 18:52:09.491901 16740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1596) took 1499 ms. Trace:
I20260810 18:52:09.501801 16622 rpcz_store.cc:276] 0810 18:52:08.155307 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:08.314000 (+158693us) service_pool.cc:227] Handling call
0810 18:52:08.359201 (+ 45201us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:08.359414 (+ 213us) raft_consensus.cc:1520] Early marking committed up to index 3
0810 18:52:09.145205 (+785791us) log.cc:842] Serialized 14 byte log entry
0810 18:52:09.500666 (+355461us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:09.500712 (+ 46us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:09.500751 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:09.500821 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:09.501016 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":58859,"prepare.queue_time_us":743008,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":25,"raft.queue_time_us":73409,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":124,"thread_start_us":1074594,"threads_started":3}
I20260810 18:52:09.503505 17787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55570 (request call id 417) took 1539 ms. Trace:
W20260810 18:52:09.507961 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 155101us
I20260810 18:52:09.502468 16740 rpcz_store.cc:276] 0810 18:52:07.992850 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:08.056757 (+ 63907us) service_pool.cc:227] Handling call
0810 18:52:08.277339 (+220582us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:08.277563 (+ 224us) raft_consensus.cc:1520] Early marking committed up to index 4
0810 18:52:08.609495 (+331932us) log.cc:842] Serialized 14 byte log entry
0810 18:52:09.491502 (+882007us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:09.491558 (+ 56us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:09.491610 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:09.491687 (+ 77us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:09.491832 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":593892,"prepare.queue_time_us":1052714,"raft.queue_time_us":719511,"raft.run_wall_time_us":3161,"spinlock_wait_cycles":22121600,"thread_start_us":573096,"threads_started":2}
I20260810 18:52:09.335754 19943 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:09.513111 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:09.520738 16628 raft_consensus.cc:3037] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:09.342638 19099 consensus_queue.cc:237] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:09.429508 19948 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:09.516919 17787 rpcz_store.cc:276] 0810 18:52:07.963894 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:07.964161 (+ 267us) service_pool.cc:227] Handling call
0810 18:52:07.964529 (+ 368us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:07.964784 (+ 255us) raft_consensus.cc:1520] Early marking committed up to index 6
0810 18:52:09.201479 (+1236695us) log.cc:842] Serialized 14 byte log entry
0810 18:52:09.502903 (+301424us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:09.503213 (+ 310us) raft_consensus.cc:1631] Marking committed up to 6
0810 18:52:09.503263 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:09.503320 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:09.503452 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":58547,"prepare.queue_time_us":1354356,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":40,"thread_start_us":1430405,"threads_started":3}
I20260810 18:52:09.424666 20057 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:09.481808 20047 raft_consensus.cc:482] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:09.529273 16749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1597) took 1536 ms. Trace:
W20260810 18:52:09.525195 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 177130us
I20260810 18:52:09.535848 16749 rpcz_store.cc:276] 0810 18:52:07.993111 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:08.188288 (+195177us) service_pool.cc:227] Handling call
0810 18:52:08.282908 (+ 94620us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:08.283127 (+ 219us) raft_consensus.cc:1520] Early marking committed up to index 5
0810 18:52:09.114313 (+831186us) log.cc:842] Serialized 14 byte log entry
0810 18:52:09.350457 (+236144us) log.cc:842] Serialized 14 byte log entry
0810 18:52:09.528825 (+178368us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:09.528930 (+ 105us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:09.528978 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:09.529072 (+ 94us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:09.529192 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12133,"prepare.queue_time_us":18551,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":20826,"spinlock_wait_cycles":4651904,"thread_start_us":384802,"threads_started":2,"wal-append.queue_time_us":305226}
I20260810 18:52:09.538352 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:09.539256 16743 raft_consensus.cc:3037] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:09.509029 19351 raft_consensus.cc:482] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:09.542117 16740 raft_consensus.cc:1261] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260810 18:52:09.544026 16636 raft_consensus.cc:1261] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260810 18:52:09.489121 19936 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:09.495143 19858 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:09.508575 18728 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:09.558691 16740 raft_consensus.cc:1261] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260810 18:52:09.576305 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:09.577678 16622 raft_consensus.cc:3037] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
W20260810 18:52:09.579248 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 438037us
I20260810 18:52:09.580391 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:09.581039 16748 raft_consensus.cc:3037] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:09.511092 20024 leader_election.cc:288] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:09.586781 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:09.587558 16745 raft_consensus.cc:3037] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
W20260810 18:52:09.591395 16701 leader_election.cc:334] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.774s (SENT)
I20260810 18:52:09.591856 16701 leader_election.cc:302] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:09.539198 20017 log.cc:924] Time spent T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.199s user 0.001s sys 0.001s
I20260810 18:52:09.599326 16628 raft_consensus.cc:2454] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:09.542985 19561 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:09.591101 20054 raft_consensus.cc:482] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:09.538445 19912 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:09.541461 19982 leader_election.cc:288] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:09.606269 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:09.626509 16631 raft_consensus.cc:3037] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:09.517345 19929 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:09.625001 16701 leader_election.cc:334] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.813s (SENT)
I20260810 18:52:09.630515 16701 leader_election.cc:302] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:09.607805 20026 log.cc:924] Time spent T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.273s user 0.000s sys 0.002s
I20260810 18:52:09.599711 19948 raft_consensus.cc:482] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:09.617372 20049 leader_election.cc:288] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:09.600548 19936 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:09.650377 20079 raft_consensus.cc:482] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:09.650785 20073 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:09.652287 16701 leader_election.cc:334] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.513s (SENT)
I20260810 18:52:09.653151 19786 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:09.654022 20056 log.cc:924] Time spent T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260810 18:52:09.584131 20064 consensus_queue.cc:1035] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20260810 18:52:09.699709 20089 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:09.700939 20012 log.cc:924] Time spent T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.173s user 0.000s sys 0.001s
I20260810 18:52:09.700703 20089 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:09.643528 19961 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:09.702543 16631 raft_consensus.cc:2454] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:09.545625 20047 raft_consensus.cc:504] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:09.714592 16743 raft_consensus.cc:2454] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
W20260810 18:52:09.657470 20033 log.cc:924] Time spent T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.301s user 0.000s sys 0.001s
I20260810 18:52:09.689463 20073 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:09.721377 19786 raft_consensus.cc:482] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:09.722121 19786 raft_consensus.cc:504] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:09.687740 20079 raft_consensus.cc:504] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:09.490862 20037 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:09.681845 16745 raft_consensus.cc:2454] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:09.588296 18728 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:09.639325 20051 log.cc:924] Time spent T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.219s user 0.000s sys 0.002s
W20260810 18:52:09.740631 19997 log.cc:924] Time spent T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 1.521s user 0.009s sys 0.013s
I20260810 18:52:09.742416 18728 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:09.587036 19858 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:09.757026 19858 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:09.754424 19987 raft_consensus.cc:504] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:09.577548 19351 raft_consensus.cc:504] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:09.765389 19351 leader_election.cc:288] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:09.674208 16622 raft_consensus.cc:2454] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:09.654681 20086 raft_consensus.cc:2726] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:52:09.620281 19561 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:09.704233 20088 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:09.705567 19765 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:09.708902 20003 log.cc:924] Time spent T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 1.341s user 0.008s sys 0.004s
I20260810 18:52:09.544745 20057 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:09.628078 20016 consensus_queue.cc:1035] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.099s
I20260810 18:52:09.774533 20037 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:09.706879 17778 consensus_meta.cc:287] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.510s user 0.012s sys 0.000s
I20260810 18:52:09.711458 19480 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:09.634794 19912 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:09.688278 19936 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:09.715631 17770 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55570 (request call id 419) took 1658 ms. Trace:
I20260810 18:52:09.717224 16746 raft_consensus.cc:1261] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260810 18:52:09.718279 19907 consensus_meta.cc:287] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.516s user 0.011s sys 0.002s
I20260810 18:52:09.625236 20054 raft_consensus.cc:504] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:09.824440 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 304448us
I20260810 18:52:09.710676 19943 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:09.735740 20079 leader_election.cc:288] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:09.752442 19786 leader_election.cc:288] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:09.744969 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:09.756422 20089 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:09.746789 20073 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:09.670936 19948 raft_consensus.cc:504] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:09.810663 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:52:09.810062 17778 raft_consensus.cc:2454] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:09.816517 19480 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:09.820997 20057 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:09.775347 19561 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:09.821697 19912 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:09.823781 19858 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
W20260810 18:52:09.732581 17726 leader_election.cc:334] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.747s (SENT)
I20260810 18:52:09.833392 17727 leader_election.cc:302] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:09.816948 20037 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:09.833559 19899 raft_consensus.cc:482] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:09.836499 19718 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:09.775759 20088 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:09.822770 20080 log.cc:924] Time spent T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.319s user 0.003s sys 0.000s
W20260810 18:52:09.839227 16701 leader_election.cc:334] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.826s (SENT)
I20260810 18:52:09.846864 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:09.850971 17726 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:09.696007 16748 raft_consensus.cc:2454] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:09.888615 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:09.888706 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 270785us
W20260810 18:52:09.841946 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 318233us
I20260810 18:52:09.887904 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:09.895262 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:09.888427 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 365638us
I20260810 18:52:09.900226 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:09.888010 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:09.898576 18728 raft_consensus.cc:482] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:09.900967 16573 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:09.888275 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 55502.2us
I20260810 18:52:09.903007 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:09.905629 16573 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:09.912410 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:09.909736 16748 raft_consensus.cc:3037] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:09.906283 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:09.912735 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:09.914743 17787 raft_consensus.cc:2454] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
I20260810 18:52:09.894122 20089 raft_consensus.cc:2781] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260810 18:52:09.940236 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 207677us
I20260810 18:52:09.948457 18728 raft_consensus.cc:504] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:09.940761 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 43571.6us
I20260810 18:52:09.916383 16746 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 5.
I20260810 18:52:09.847295 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:09.851835 19929 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:09.853117 17778 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55570 (request call id 420) took 1317 ms. Trace:
I20260810 18:52:09.882081 19987 leader_election.cc:288] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:09.951359 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:09.906113 20101 raft_consensus.cc:2726] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260810 18:52:09.956851 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 117636us
I20260810 18:52:09.957398 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:09.899240 17781 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55602 (request call id 331) took 1969 ms. Trace:
I20260810 18:52:09.903555 19907 raft_consensus.cc:482] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:09.939802 19595 raft_consensus.cc:979] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be: : Instructing follower d304ac47fdf846de81337057a4ef7bea to start an election
I20260810 18:52:09.903470 19786 raft_consensus.cc:2726] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:52:09.899818 20062 raft_consensus.cc:2726] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:52:09.904255 19862 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:09.943886 20071 raft_consensus.cc:482] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:09.975960 20120 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:09.976853 20096 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
W20260810 18:52:09.957805 16701 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.887s (SENT) [suppressed 38 similar messages]
I20260810 18:52:09.870977 19858 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:09.877595 19899 raft_consensus.cc:504] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:09.963943 17781 rpcz_store.cc:276] 0810 18:52:07.929552 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:07.941498 (+ 11946us) service_pool.cc:227] Handling call
0810 18:52:07.956469 (+ 14971us) raft_consensus.cc:1457] Updating replica for 2 ops
0810 18:52:07.956908 (+ 439us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:07.956938 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0810 18:52:07.957541 (+ 603us) log.cc:842] Serialized 52 byte log entry
0810 18:52:09.707347 (+1749806us) spinlock_profiling.cc:243] Waited 15 ms on lock 0x7b44000232cc. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae50e4a95 00007f1ae50e54b2 00007f1ae7d8d511 00007f1ae7d88e1a 00007f1ae7d7cbee 00007f1ae7d7bf36 00007f1ae7d7bb58 00007f1ae7d85551 00007f1ae7d85509 00007f1ae7d85499
0810 18:52:09.760613 (+ 53266us) spinlock_profiling.cc:243] Waited 7.89 ms on lock 0x7b18000be180. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1aee0718b1 00007f1aee07e661 00007f1ae7d8982c 00007f1ae51b49dc 00007f1aedfeba39 00007f1ae51d18bb 00007f1ae51cfcbf 00007f1ae51d1e22 00007f1ae7d7b5bb 00007f1ae7d81063 00007f1ae7d81411
0810 18:52:09.762541 (+ 1928us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:09.762662 (+ 121us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:09.762806 (+ 144us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:09.887936 (+125130us) raft_consensus.cc:1671] finished
0810 18:52:09.889134 (+ 1198us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:09.896521 (+ 7387us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":405,"raft.queue_time_us":7533,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":37945,"spinlock_wait_cycles":52678656,"thread_start_us":96683,"threads_started":1,"wal-append.queue_time_us":409043}
I20260810 18:52:10.028865 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:09.947923 20089 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:10.029867 20089 raft_consensus.cc:3037] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:09.869620 20037 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:09.885783 19718 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:09.972885 17775 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:10.009055 20092 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:09.973026 19765 raft_consensus.cc:2726] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:09.981843 19929 raft_consensus.cc:1067] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Signalling peer d304ac47fdf846de81337057a4ef7bea to start an election
I20260810 18:52:10.016932 17785 raft_consensus.cc:1261] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260810 18:52:10.024034 19480 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:10.007372 19561 consensus_queue.cc:1035] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260810 18:52:10.088573 20037 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:10.040726 16748 raft_consensus.cc:2454] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 4.
I20260810 18:52:09.961659 17778 rpcz_store.cc:276] 0810 18:52:08.532988 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:08.600569 (+ 67581us) service_pool.cc:227] Handling call
0810 18:52:09.850513 (+1249944us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:10.090057 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:10.090907 17778 raft_consensus.cc:2454] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:10.092478 17778 raft_consensus.cc:1261] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260810 18:52:10.042277 16573 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
W20260810 18:52:10.093664 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 51509us
I20260810 18:52:10.094307 19718 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:10.095139 20113 raft_consensus.cc:2781] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:10.095613 20113 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:10.095958 20113 raft_consensus.cc:3037] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:10.058617 18728 leader_election.cc:288] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:10.068660 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 86212.2us
I20260810 18:52:10.074554 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:09.817168 17770 rpcz_store.cc:276] 0810 18:52:08.057228 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:08.073410 (+ 16182us) service_pool.cc:227] Handling call
0810 18:52:08.155977 (+ 82567us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:08.289620 (+133643us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:08.292008 (+ 2388us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:08.379198 (+ 87190us) log.cc:842] Serialized 26 byte log entry
0810 18:52:09.248500 (+869302us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:09.248550 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:09.248630 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:09.715206 (+466576us) raft_consensus.cc:1671] finished
0810 18:52:09.715225 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:09.715523 (+ 298us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":153,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":6523,"thread_start_us":868663,"threads_started":1,"wal-append.queue_time_us":939200}
I20260810 18:52:10.074915 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:10.075132 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:10.075423 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:10.075831 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:10.076510 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:10.077241 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:09.978792 19907 raft_consensus.cc:504] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:09.997552 19862 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:10.080020 19948 leader_election.cc:288] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:10.057634 20120 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:10.009863 19912 consensus_queue.cc:1035] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.117s
I20260810 18:52:10.039006 20096 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:10.211529 16752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 192) took 1974 ms. Trace:
I20260810 18:52:10.220849 16496 raft_consensus.cc:3037] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 4
I20260810 18:52:10.197678 16450 leader_election.cc:302] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:10.223620 16450 leader_election.cc:302] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:10.129962 16447 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:10.153069 19595 consensus_queue.cc:1035] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.079s
W20260810 18:52:10.233933 17726 leader_election.cc:334] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.026s (SENT)
I20260810 18:52:10.012125 20071 raft_consensus.cc:504] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:10.153990 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 196204us
W20260810 18:52:10.238157 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 301414us
W20260810 18:52:10.239672 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 109856us
I20260810 18:52:10.243839 16450 leader_election.cc:302] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:10.221011 16752 rpcz_store.cc:276] 0810 18:52:08.236819 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:08.296235 (+ 59416us) service_pool.cc:227] Handling call
0810 18:52:08.344544 (+ 48309us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:09.254031 (+909487us) raft_consensus.cc:1520] Early marking committed up to index 5
0810 18:52:09.493745 (+239714us) log.cc:842] Serialized 14 byte log entry
0810 18:52:09.888287 (+394542us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:09.893090 (+ 4803us) log.cc:842] Serialized 26 byte log entry
0810 18:52:09.894091 (+ 1001us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:09.894142 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:09.894232 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:10.211227 (+316995us) raft_consensus.cc:1671] finished
0810 18:52:10.211255 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:10.211446 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12446,"prepare.queue_time_us":76725,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":8869,"raft.queue_time_us":887748,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":33471,"thread_start_us":720037,"threads_started":3,"wal-append.queue_time_us":485389}
W20260810 18:52:10.241708 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 289766us
I20260810 18:52:10.240727 18728 consensus_queue.cc:1035] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:52:10.243104 16447 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:10.249827 18728 raft_consensus.cc:2781] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:10.250389 18728 raft_consensus.cc:686] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:10.099730 20088 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:10.263128 17787 raft_consensus.cc:2396] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260810 18:52:10.212170 19765 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:10.266744 17727 leader_election.cc:302] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:10.248142 17771 raft_consensus.cc:2396] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260810 18:52:10.181288 19718 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:10.271869 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:10.272802 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:10.280277 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:10.286079 16743 raft_consensus.cc:3037] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:10.286700 16746 raft_consensus.cc:1261] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260810 18:52:10.244191 16701 connection.cc:232] RPC call timeout handler was delayed by -1.9156494119670242s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3022s for a response
W20260810 18:52:10.289777 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 46771us
I20260810 18:52:10.271098 16748 raft_consensus.cc:1261] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260810 18:52:10.070484 20054 leader_election.cc:288] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:10.154237 20037 consensus_queue.cc:1035] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.131s
I20260810 18:52:10.126576 20092 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:10.343880 16749 raft_consensus.cc:2454] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:10.500998 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:10.350539 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:10.297328 16750 raft_consensus.cc:1261] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260810 18:52:10.341187 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 84554.1us
I20260810 18:52:10.547968 16743 raft_consensus.cc:2454] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:10.376103 16501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1435) took 2853 ms. Trace:
I20260810 18:52:10.598680 16496 raft_consensus.cc:2454] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 4.
I20260810 18:52:10.247234 19561 raft_consensus.cc:2781] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election won for term 3
W20260810 18:52:10.287792 16701 connection.cc:232] RPC call timeout handler was delayed by -1.8532651399727911s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20320000000000002s for a response
W20260810 18:52:10.652801 16701 connection.cc:232] RPC call timeout handler was delayed by -1.8436736250296235s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2403s for a response
I20260810 18:52:10.349200 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:10.292622 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:10.653985 16748 raft_consensus.cc:2454] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
I20260810 18:52:10.293431 18728 consensus_queue.cc:237] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:10.655081 16701 connection.cc:232] RPC call timeout handler was delayed by -1.7911528060212731s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.217s for a response
W20260810 18:52:10.655398 16701 connection.cc:232] RPC call timeout handler was delayed by -1.7902528059203178s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2171s for a response
W20260810 18:52:10.655701 16701 connection.cc:232] RPC call timeout handler was delayed by -1.7623651400208473s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21330000000000002s for a response
W20260810 18:52:10.656153 16701 leader_election.cc:334] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.616s (SENT)
I20260810 18:52:10.644313 16501 rpcz_store.cc:276] 0810 18:52:07.522169 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:07.522298 (+ 129us) service_pool.cc:227] Handling call
0810 18:52:07.522649 (+ 351us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:08.408756 (+886107us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:10.375794 (+1967038us) spinlock_profiling.cc:243] Waited 827 ms on lock 0x7b5400116a54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:10.376000 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1901501824}
I20260810 18:52:10.635917 16752 raft_consensus.cc:2454] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:52:10.656754 16701 leader_election.cc:302] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:10.656070 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:10.488085 20079 raft_consensus.cc:482] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:10.668239 20079 raft_consensus.cc:504] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
W20260810 18:52:10.668848 20135 log.cc:924] Time spent T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260810 18:52:10.669891 20147 raft_consensus.cc:482] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:10.671861 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:10.672593 16749 raft_consensus.cc:2454] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 4.
W20260810 18:52:10.371873 20115 log.cc:924] Time spent T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.190s user 0.001s sys 0.001s
I20260810 18:52:10.673732 16495 raft_consensus.cc:2454] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 3.
I20260810 18:52:10.358557 19858 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:10.729411 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:10.297693 20088 raft_consensus.cc:2781] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:52:10.296939 19289 raft_consensus.cc:2781] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:10.452935 19907 leader_election.cc:288] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:10.362866 19976 raft_consensus.cc:2781] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:10.699725 17727 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:10.371156 20142 raft_consensus.cc:2781] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:52:10.493642 19862 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:10.605484 20071 leader_election.cc:288] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:10.373313 20096 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:10.671730 19948 raft_consensus.cc:2726] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260810 18:52:10.742538 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 86402us
W20260810 18:52:10.758046 16701 leader_election.cc:334] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.654s (SENT)
I20260810 18:52:10.758558 16701 leader_election.cc:302] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:10.502123 20089 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:10.435083 20113 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:10.299481 19718 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:10.737629 19289 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:10.741488 20120 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:10.791383 16701 leader_election.cc:334] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.903s (SENT)
W20260810 18:52:10.777179 17726 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.021s sys 0.065s
I20260810 18:52:10.745972 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:10.791939 16701 leader_election.cc:302] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:10.792524 17726 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.556s user 0.022s sys 0.066s
W20260810 18:52:10.794181 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 558185us
I20260810 18:52:10.670573 20147 raft_consensus.cc:504] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:10.295535 19765 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:10.776997 19976 raft_consensus.cc:686] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:10.766942 20142 raft_consensus.cc:686] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:10.789623 19289 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
W20260810 18:52:10.836083 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 44748.7us
W20260810 18:52:10.842984 16701 leader_election.cc:334] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.058s (SENT)
I20260810 18:52:10.844733 16701 leader_election.cc:302] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:10.763118 20095 consensus_queue.cc:1035] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
I20260810 18:52:10.822835 19948 raft_consensus.cc:2726] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:52:10.827924 20126 raft_consensus.cc:2726] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:52:10.828899 20079 leader_election.cc:288] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:10.866289 17781 raft_consensus.cc:2379] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 5: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
W20260810 18:52:10.888612 20148 log.cc:924] Time spent T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.129s user 0.001s sys 0.000s
W20260810 18:52:10.890273 20157 log.cc:924] Time spent T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
I20260810 18:52:10.868001 19858 raft_consensus.cc:2781] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:10.878499 19948 raft_consensus.cc:2726] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260810 18:52:10.922273 16747 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.006s sys 0.026s
W20260810 18:52:10.925967 16747 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.563s user 0.006s sys 0.026s
I20260810 18:52:10.870257 20110 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:10.883821 19561 raft_consensus.cc:686] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:10.919554 20088 raft_consensus.cc:686] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 4 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:10.937983 20088 consensus_queue.cc:237] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:10.940828 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 97885.6us
W20260810 18:52:10.941622 16701 leader_election.cc:334] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.061s (SENT)
I20260810 18:52:10.942310 16701 leader_election.cc:302] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:10.944023 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 271762us
I20260810 18:52:10.924343 20079 raft_consensus.cc:482] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:10.952240 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:10.953189 16746 raft_consensus.cc:3037] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
W20260810 18:52:10.954384 20154 log.cc:924] Time spent T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.125s user 0.000s sys 0.001s
I20260810 18:52:10.955854 19835 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:10.956072 20175 log.cc:924] Time spent T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.064s user 0.000s sys 0.001s
I20260810 18:52:10.944728 17786 raft_consensus.cc:2932] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 2 to 4, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:52:10.948654 19948 raft_consensus.cc:2726] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:10.977754 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:10.968853 18728 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:10.980940 18728 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:10.983368 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:10.985597 20070 log.cc:924] Time spent T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 1.564s user 0.004s sys 0.020s
I20260810 18:52:10.952400 20092 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:10.976897 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:10.942325 20142 consensus_queue.cc:237] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:10.989179 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:10.975903 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:10.952183 20089 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:10.970217 16701 leader_election.cc:334] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.062s (SENT)
I20260810 18:52:10.999284 16701 leader_election.cc:302] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:11.002152 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:11.002177 18728 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:11.004033 16623 raft_consensus.cc:1261] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260810 18:52:11.009436 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:11.010214 16623 raft_consensus.cc:3037] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:11.018108 16746 raft_consensus.cc:2454] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 5.
I20260810 18:52:11.025120 17727 leader_election.cc:302] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:11.025640 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:11.027549 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:11.032835 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 62650.2us
W20260810 18:52:11.033704 16701 leader_election.cc:334] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.169s (SENT)
I20260810 18:52:11.034520 20089 raft_consensus.cc:2781] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:11.035130 20089 raft_consensus.cc:686] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 5 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:10.956470 19835 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:11.040616 16701 leader_election.cc:302] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:11.036038 16504 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7ef7c65533724e678b75830002906e2c"
dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
from {username='slave'} at 127.0.0.1:52352
I20260810 18:52:11.078399 16504 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:52:11.078927 16504 raft_consensus.cc:3037] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:10.965135 20079 raft_consensus.cc:504] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:11.082574 20147 leader_election.cc:288] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:11.161703 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:11.162036 19934 leader_election.cc:288] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:11.162631 19934 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:11.162938 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:11.163147 19934 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:11.094717 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:11.168879 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:11.170269 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:11.172982 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:11.095371 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:11.094871 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:11.184558 16623 raft_consensus.cc:2454] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:10.932000 20110 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:11.036952 19948 raft_consensus.cc:2726] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:52:10.930437 19976 consensus_queue.cc:237] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:11.219394 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:10.967056 19561 consensus_queue.cc:237] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:11.219693 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 186027us
W20260810 18:52:11.244284 16701 connection.cc:232] RPC call timeout handler was delayed by -1.4698651400394738s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24580000000000002s for a response
W20260810 18:52:11.244767 16701 connection.cc:232] RPC call timeout handler was delayed by -1.4602736250963062s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.28290000000000004s for a response
W20260810 18:52:11.245442 16701 connection.cc:232] RPC call timeout handler was delayed by -1.2793494120705873s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3729s for a response
W20260810 18:52:11.245849 16701 connection.cc:232] RPC call timeout handler was delayed by -1.17461930308491s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16290000000000002s for a response
W20260810 18:52:11.246104 16701 connection.cc:232] RPC call timeout handler was delayed by -1.170119303278625s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1634s for a response
W20260810 18:52:11.246374 16701 connection.cc:232] RPC call timeout handler was delayed by -1.0761469590943307s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2096s for a response
I20260810 18:52:11.248631 16622 raft_consensus.cc:2454] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
W20260810 18:52:11.250239 16701 leader_election.cc:334] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.582s (SENT)
W20260810 18:52:11.251597 16701 leader_election.cc:334] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.629s (SENT)
W20260810 18:52:11.252696 16701 leader_election.cc:334] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.634s (SENT)
I20260810 18:52:11.182083 16746 raft_consensus.cc:2454] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
W20260810 18:52:11.110594 19963 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 3.115s user 0.014s sys 0.053s
W20260810 18:52:11.263721 19963 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 3.268s user 0.014s sys 0.062s
I20260810 18:52:11.100281 20126 raft_consensus.cc:2726] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:11.227542 16450 leader_election.cc:302] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:11.228081 20079 leader_election.cc:288] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:11.093470 19765 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:11.093730 20113 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:11.238402 16508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:52330 (request call id 805) took 3281 ms. Trace:
I20260810 18:52:11.239164 20184 consensus_queue.cc:1035] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
I20260810 18:52:11.243216 16631 raft_consensus.cc:1261] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260810 18:52:11.260169 19948 raft_consensus.cc:482] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:11.265120 16450 leader_election.cc:302] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:11.179370 19945 consensus_queue.cc:1035] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.064s
I20260810 18:52:11.231127 19934 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:11.269258 16508 rpcz_store.cc:276] 0810 18:52:07.948285 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:07.959179 (+ 10894us) service_pool.cc:227] Handling call
0810 18:52:11.228941 (+3269762us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b54000f7154. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807faf0 00007f1ae8102f2a 00007f1aed03fc6e 00007f1ae6736807 00007f1ae6736792 00007f1ae67366c1 00007f1ae673663c 00007f1ae67358f2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:11.229557 (+ 616us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3755193216}
I20260810 18:52:11.277566 16450 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:11.281008 16701 leader_election.cc:334] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.941s (SENT)
I20260810 18:52:11.282857 16701 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:11.285804 16450 leader_election.cc:302] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:11.287597 20181 raft_consensus.cc:482] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:11.280189 20092 raft_consensus.cc:2781] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:11.288443 20092 raft_consensus.cc:686] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:11.270669 19982 raft_consensus.cc:2781] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:11.219108 20089 consensus_queue.cc:237] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:11.303159 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:11.295264 20185 log.cc:924] Time spent T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.066s user 0.001s sys 0.000s
I20260810 18:52:11.295708 20172 raft_consensus.cc:482] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:11.297668 19936 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:11.302125 16626 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1408) took 2957 ms (client timeout 1662 ms). Trace:
I20260810 18:52:11.280652 20047 leader_election.cc:288] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:11.288120 20181 raft_consensus.cc:504] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:11.306676 16631 raft_consensus.cc:2396] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260810 18:52:11.303431 20126 raft_consensus.cc:2726] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260810 18:52:11.365442 16447 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.024s sys 0.027s
W20260810 18:52:11.384159 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 103169us
I20260810 18:52:11.464839 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:11.464617 20092 consensus_queue.cc:237] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:11.466234 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:11.442240 16504 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:11.467006 16631 raft_consensus.cc:2362] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4. Re-sending same reply.
I20260810 18:52:11.468314 16504 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:11.466259 16701 leader_election.cc:334] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.096s (SENT)
I20260810 18:52:11.288506 19934 consensus_queue.cc:1035] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260810 18:52:11.414839 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:11.291656 19982 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
W20260810 18:52:11.464591 16447 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.634s user 0.024s sys 0.037s
W20260810 18:52:11.468858 16626 rpcz_store.cc:269] 0810 18:52:08.344540 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:11.301934 (+2957394us) service_pool.cc:214] Skipping call since client already timed out
0810 18:52:11.302038 (+ 104us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260810 18:52:11.468554 16450 leader_election.cc:302] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:11.283718 20095 raft_consensus.cc:2781] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:11.289048 19907 raft_consensus.cc:2781] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:52:11.281031 19948 raft_consensus.cc:504] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:11.480710 16448 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:11.491686 16701 leader_election.cc:302] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:11.492003 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:11.438961 19835 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:11.282035 19289 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:11.512771 16448 leader_election.cc:302] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:52:11.529793 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 63560.8us
I20260810 18:52:11.526187 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:11.531265 16631 raft_consensus.cc:2354] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 4. Current term is 5.
W20260810 18:52:11.531132 16701 leader_election.cc:334] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.170s (SENT)
I20260810 18:52:11.531821 16701 leader_election.cc:302] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:11.514285 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 45807us
I20260810 18:52:11.489048 17775 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55586 (request call id 328) took 1227 ms. Trace:
I20260810 18:52:11.514341 16622 raft_consensus.cc:2454] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:11.528115 19976 raft_consensus.cc:2726] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:11.540999 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:11.546339 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:11.559651 16631 raft_consensus.cc:2354] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 4. Current term is 5.
I20260810 18:52:11.560688 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:11.541671 16746 raft_consensus.cc:1261] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260810 18:52:11.568064 16501 raft_consensus.cc:1226] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:11.569065 16624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1579) took 2028 ms. Trace:
I20260810 18:52:11.569658 16747 raft_consensus.cc:1261] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260810 18:52:11.569439 16624 rpcz_store.cc:276] 0810 18:52:09.540994 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:11.534709 (+1993715us) service_pool.cc:227] Handling call
0810 18:52:11.568559 (+ 33850us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:11.568750 (+ 191us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:11.568772 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:11.568809 (+ 37us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:11.568849 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:11.568901 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:11.569009 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:11.569722 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:11.577729 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:11.578653 16746 raft_consensus.cc:1226] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260810 18:52:11.510808 19907 raft_consensus.cc:482] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:11.526391 19982 raft_consensus.cc:3037] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:11.512261 20126 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:11.578672 16631 raft_consensus.cc:2454] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:11.579658 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:11.579957 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:11.513034 20095 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:11.534682 19835 raft_consensus.cc:2726] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:11.584250 17786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55570 (request call id 418) took 3616 ms. Trace:
I20260810 18:52:11.581248 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:11.584534 20095 raft_consensus.cc:3037] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:11.585196 16622 raft_consensus.cc:2379] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 2: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:11.483995 20172 raft_consensus.cc:504] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:11.586052 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:11.725494 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:11.726634 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:11.586824 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:11.731006 16624 raft_consensus.cc:2454] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
I20260810 18:52:11.618435 17787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55586 (request call id 322) took 1666 ms. Trace:
I20260810 18:52:11.732074 17787 rpcz_store.cc:276] 0810 18:52:09.952417 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:09.952550 (+ 133us) service_pool.cc:227] Handling call
0810 18:52:11.618224 (+1665674us) spinlock_profiling.cc:243] Waited 305 ms on lock 0x7b54006a27d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:11.618373 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":700439424}
I20260810 18:52:11.584267 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:11.734261 16626 raft_consensus.cc:2454] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
W20260810 18:52:11.603973 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 72888.7us
I20260810 18:52:11.736788 16622 raft_consensus.cc:2379] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 3: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:11.559226 20047 raft_consensus.cc:2726] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:52:11.586735 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:11.584654 17786 rpcz_store.cc:276] 0810 18:52:07.967870 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:07.967999 (+ 129us) service_pool.cc:227] Handling call
0810 18:52:08.642942 (+674943us) spinlock_profiling.cc:243] Waited 27 ms on lock 0x7b5800005498. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed0502b0 00007f1aed03b7f9 00007f1aed03b2a2 00007f1ae6721684
0810 18:52:10.942897 (+2299955us) spinlock_profiling.cc:243] Waited 4.88 ms on lock 0x7b5800005498. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519ab7c 00007f1ae519a73b 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed0502b0 00007f1aed03b7f9 00007f1aed03b2a2 00007f1ae6721684
0810 18:52:10.943145 (+ 248us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:10.944567 (+ 1422us) raft_consensus.cc:1520] Early marking committed up to index 5
0810 18:52:11.045957 (+101390us) log.cc:842] Serialized 14 byte log entry
0810 18:52:11.583618 (+537661us) log.cc:842] Serialized 14 byte log entry
0810 18:52:11.583814 (+ 196us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:11.583858 (+ 44us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:11.583901 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:11.583959 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:11.584074 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":40751,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":58,"raft.queue_time_us":2729,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":3718,"spinlock_wait_cycles":145311104,"thread_start_us":487990,"threads_started":1,"wal-append.queue_time_us":671505}
I20260810 18:52:11.587219 16745 raft_consensus.cc:2454] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:11.738001 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:11.751257 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:11.751938 16701 leader_election.cc:334] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.458s (SENT)
I20260810 18:52:11.752306 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:11.753100 16624 raft_consensus.cc:2454] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:11.752424 16701 leader_election.cc:302] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:11.735970 19961 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:11.755174 16626 raft_consensus.cc:2379] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 4: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:11.598145 19907 raft_consensus.cc:3037] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
W20260810 18:52:11.756623 16701 leader_election.cc:334] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 3.063s (SENT)
I20260810 18:52:11.757817 16633 raft_consensus.cc:2354] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 4. Current term is 5.
I20260810 18:52:11.741191 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:11.708357 19289 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:11.613401 19948 leader_election.cc:288] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:11.758773 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:11.759198 19961 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:11.768069 16701 leader_election.cc:302] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:11.759851 20200 log.cc:924] Time spent T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260810 18:52:11.613067 20067 raft_consensus.cc:2726] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:11.758014 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:11.774262 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:11.774816 16622 raft_consensus.cc:2379] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 4: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:11.774377 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:11.761353 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:11.775934 16624 raft_consensus.cc:2354] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3.
W20260810 18:52:11.773731 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 137284us
W20260810 18:52:11.767298 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 234933us
I20260810 18:52:11.762382 16631 raft_consensus.cc:2454] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:11.769340 16633 raft_consensus.cc:2362] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2. Re-sending same reply.
I20260810 18:52:11.772107 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:11.773232 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 111279us
I20260810 18:52:11.784489 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:11.785158 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:11.757403 20047 raft_consensus.cc:2726] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:11.787632 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:11.787797 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:11.612004 20124 raft_consensus.cc:2781] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:11.789361 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:11.791414 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:11.792196 16624 raft_consensus.cc:2454] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
I20260810 18:52:11.792281 19029 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:11.793277 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:11.793033 19029 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:11.794567 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:11.611799 20134 consensus_queue.cc:1035] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.288s
I20260810 18:52:11.617436 20181 leader_election.cc:288] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:11.789038 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:11.861696 16622 raft_consensus.cc:3037] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:11.795768 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:11.785797 16633 raft_consensus.cc:2454] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:11.862773 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:11.863265 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:52:11.846596 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 49222.3us
I20260810 18:52:11.850128 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:52:11.846050 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 59264.3us
I20260810 18:52:11.848232 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:11.864537 16448 leader_election.cc:302] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:11.604682 20126 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:11.537714 17775 rpcz_store.cc:276] 0810 18:52:10.261046 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:10.284329 (+ 23283us) service_pool.cc:227] Handling call
0810 18:52:10.358848 (+ 74519us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:10.375618 (+ 16770us) raft_consensus.cc:1520] Early marking committed up to index 5
0810 18:52:10.762752 (+387134us) log.cc:842] Serialized 14 byte log entry
0810 18:52:10.963287 (+200535us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:11.096477 (+133190us) log.cc:842] Serialized 26 byte log entry
0810 18:52:11.179611 (+ 83134us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:11.189005 (+ 9394us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:11.221819 (+ 32814us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:11.454007 (+232188us) raft_consensus.cc:1671] finished
0810 18:52:11.454036 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:11.488289 (+ 34253us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":677,"prepare.queue_time_us":492177,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":48,"raft.queue_time_us":3213,"raft.run_cpu_time_us":345,"raft.run_wall_time_us":6800,"thread_start_us":480884,"threads_started":2,"wal-append.queue_time_us":292729}
I20260810 18:52:11.785508 19948 raft_consensus.cc:2726] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:11.783234 20062 raft_consensus.cc:482] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:11.866575 20062 raft_consensus.cc:504] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:11.868191 20126 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:11.865496 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:11.868862 20062 leader_election.cc:288] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:11.869841 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:11.875025 16631 raft_consensus.cc:1261] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260810 18:52:12.062106 16632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1584) took 1092 ms. Trace:
I20260810 18:52:11.924786 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:11.877172 16626 raft_consensus.cc:1226] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 4 LEADER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:11.882511 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:11.879698 16633 raft_consensus.cc:2379] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 3: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
W20260810 18:52:12.061543 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 197102us
I20260810 18:52:11.905537 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:12.061162 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:11.945830 16622 raft_consensus.cc:2454] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:12.100322 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:12.101599 16622 raft_consensus.cc:2379] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 3: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:11.795616 20124 raft_consensus.cc:482] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
W20260810 18:52:11.862329 20194 log.cc:924] Time spent T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.195s user 0.001s sys 0.000s
I20260810 18:52:11.861130 19279 consensus_queue.cc:1035] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.351s
I20260810 18:52:12.214574 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:11.918265 19391 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:12.216481 19391 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:12.105449 16447 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:52:11.879206 16693 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.000s sys 0.037s
I20260810 18:52:12.064324 16632 rpcz_store.cc:276] 0810 18:52:10.970054 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:10.975491 (+ 5437us) service_pool.cc:227] Handling call
0810 18:52:11.007023 (+ 31532us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:11.022701 (+ 15678us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:11.022749 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:11.092088 (+ 69339us) log.cc:842] Serialized 26 byte log entry
0810 18:52:11.337933 (+245845us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:52:11.337990 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:11.338092 (+ 102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:12.061792 (+723700us) raft_consensus.cc:1671] finished
0810 18:52:12.061825 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:12.062040 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12109,"raft.queue_time_us":17297,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":47891,"thread_start_us":117442,"threads_started":1,"wal-append.queue_time_us":486742}
W20260810 18:52:12.218472 16693 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.050s user 0.008s sys 0.052s
I20260810 18:52:12.137285 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:12.219455 16632 raft_consensus.cc:1261] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260810 18:52:12.136353 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:12.220767 20206 log.cc:924] Time spent T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.310s user 0.000s sys 0.000s
I20260810 18:52:12.224804 17770 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55586 (request call id 330) took 1464 ms. Trace:
I20260810 18:52:12.060141 19961 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:12.229909 20190 consensus_queue.cc:1035] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:52:12.139843 20172 leader_election.cc:288] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:11.949118 20198 log.cc:924] Time spent T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.202s user 0.002s sys 0.000s
W20260810 18:52:11.924383 20204 log.cc:924] Time spent T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.200s user 0.001s sys 0.000s
I20260810 18:52:12.211526 20124 raft_consensus.cc:3037] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:12.068442 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:11.917909 19835 raft_consensus.cc:2781] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:12.400498 19835 raft_consensus.cc:482] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:12.401134 19835 raft_consensus.cc:3037] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:12.054466 19029 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:12.219020 20199 log.cc:924] Time spent T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.210s user 0.000s sys 0.001s
W20260810 18:52:11.902598 20187 log.cc:924] Time spent T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.089s user 0.001s sys 0.000s
I20260810 18:52:12.505012 19987 raft_consensus.cc:2726] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:11.911587 20171 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:12.225358 17770 rpcz_store.cc:276] 0810 18:52:10.760679 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:10.846765 (+ 86086us) service_pool.cc:227] Handling call
0810 18:52:10.926576 (+ 79811us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:10.951036 (+ 24460us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:10.962398 (+ 11362us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:11.079689 (+117291us) log.cc:842] Serialized 26 byte log entry
0810 18:52:11.572286 (+492597us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:11.580479 (+ 8193us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:11.600142 (+ 19663us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:12.224483 (+624341us) raft_consensus.cc:1671] finished
0810 18:52:12.224529 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:12.224723 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5280,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":5894,"thread_start_us":307835,"threads_started":1,"wal-append.queue_time_us":579904}
W20260810 18:52:12.517546 16750 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.738s user 0.011s sys 0.029s
W20260810 18:52:12.489216 19907 consensus_meta.cc:287] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.714s user 0.005s sys 0.006s
W20260810 18:52:12.492648 20095 consensus_meta.cc:287] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.904s user 0.011s sys 0.000s
I20260810 18:52:12.311036 20211 raft_consensus.cc:482] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:11.911777 19976 consensus_queue.cc:1035] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.577s
I20260810 18:52:12.549180 20211 raft_consensus.cc:504] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:12.567924 20211 leader_election.cc:288] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:12.569777 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:12.571441 16701 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.714s user 0.002s sys 0.007s
W20260810 18:52:12.571781 16701 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.714s user 0.002s sys 0.007s
W20260810 18:52:12.572394 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 467087us
W20260810 18:52:12.518002 16750 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.738s user 0.011s sys 0.029s
I20260810 18:52:12.546275 20095 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:12.613101 16573 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.013s sys 0.000s
W20260810 18:52:12.613724 16573 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.510s user 0.013s sys 0.000s
W20260810 18:52:12.614105 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 511000us
W20260810 18:52:12.615897 16448 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.000s sys 0.013s
W20260810 18:52:12.616401 16448 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.511s user 0.000s sys 0.014s
W20260810 18:52:12.616780 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 511186us
I20260810 18:52:12.617661 16448 leader_election.cc:302] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:52:12.631121 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 50024.3us
I20260810 18:52:12.632088 16450 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:12.633411 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 57891.7us
W20260810 18:52:12.720052 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 102500us
I20260810 18:52:12.548296 19907 raft_consensus.cc:504] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:12.517946 20171 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:12.587760 20134 consensus_queue.cc:1046] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.117s
I20260810 18:52:12.648764 20211 raft_consensus.cc:482] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:12.794355 19244 raft_consensus.cc:2781] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260810 18:52:12.723762 19982 consensus_meta.cc:287] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 1.119s user 0.011s sys 0.003s
I20260810 18:52:12.808101 20190 raft_consensus.cc:2781] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:12.802321 19982 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:12.808591 20190 raft_consensus.cc:482] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:12.809224 20190 raft_consensus.cc:3037] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:12.807235 20211 raft_consensus.cc:504] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:12.810015 20188 log.cc:924] Time spent T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 1.370s user 0.007s sys 0.020s
I20260810 18:52:12.811889 20095 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:12.755079 20172 raft_consensus.cc:482] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:12.813511 16496 raft_consensus.cc:1226] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 2. Current term is 3. Ops: []
I20260810 18:52:12.802084 19244 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:12.813730 16750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1643) took 1272 ms. Trace:
I20260810 18:52:12.823958 19244 raft_consensus.cc:3037] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:12.854882 16508 raft_consensus.cc:1226] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:12.822932 16750 rpcz_store.cc:276] 0810 18:52:11.541574 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:11.578974 (+ 37400us) service_pool.cc:227] Handling call
0810 18:52:11.612420 (+ 33446us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:11.612786 (+ 366us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:11.612815 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:11.751955 (+139140us) log.cc:842] Serialized 26 byte log entry
0810 18:52:12.586093 (+834138us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:12.586145 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:12.586241 (+ 96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:12.813504 (+227263us) raft_consensus.cc:1671] finished
0810 18:52:12.813522 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:12.813666 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3452,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":140,"thread_start_us":805950,"threads_started":1,"wal-append.queue_time_us":1056077}
I20260810 18:52:12.824604 20172 raft_consensus.cc:504] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:12.965644 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 586094us
I20260810 18:52:12.967762 16501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1445) took 1994 ms. Trace:
I20260810 18:52:12.968221 16501 rpcz_store.cc:276] 0810 18:52:10.973034 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:10.973199 (+ 165us) service_pool.cc:227] Handling call
0810 18:52:10.973594 (+ 395us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:11.568490 (+594896us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:12.967528 (+1399038us) spinlock_profiling.cc:243] Waited 594 ms on lock 0x7b5400137ed4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:12.967707 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1366666112}
I20260810 18:52:12.967808 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:12.974551 17726 leader_election.cc:334] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.935s (SENT)
I20260810 18:52:12.983526 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:12.983397 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:12.992372 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:12.995786 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:12.874207 20089 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.643s user 0.003s sys 0.013s
I20260810 18:52:12.998056 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:12.999241 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:12.890130 20171 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:12.960036 20067 raft_consensus.cc:482] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:12.961767 19907 leader_election.cc:288] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:13.005512 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:13.005616 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:13.010067 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:13.010233 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:52:13.002269 20089 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.771s user 0.004s sys 0.014s
W20260810 18:52:13.015379 16701 leader_election.cc:334] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 3.130s (SENT)
I20260810 18:52:13.030691 20211 leader_election.cc:288] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:13.031430 20211 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:13.032027 20211 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:13.011637 19835 consensus_meta.cc:287] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.610s user 0.009s sys 0.004s
I20260810 18:52:13.034802 20211 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:13.035607 20124 consensus_meta.cc:287] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.531s user 0.011s sys 0.000s
I20260810 18:52:13.003088 20190 raft_consensus.cc:504] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:13.018550 20233 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:13.022857 20047 raft_consensus.cc:482] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:13.005776 19244 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:13.012048 20067 raft_consensus.cc:504] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:13.058374 19835 raft_consensus.cc:504] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:13.124395 20068 log.cc:924] Time spent T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 3.460s user 0.018s sys 0.038s
I20260810 18:52:13.156471 20233 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:13.065295 19391 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:13.374423 20238 raft_consensus.cc:482] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:13.371939 16443 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.500s user 0.003s sys 0.012s
I20260810 18:52:13.394061 20256 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:13.167443 20248 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:13.485901 20256 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:13.486054 20248 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:13.487102 19835 leader_election.cc:288] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:13.488448 20256 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:13.489091 16443 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.998s user 0.007s sys 0.025s
I20260810 18:52:13.488833 20248 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:13.343298 20211 raft_consensus.cc:482] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:13.489971 20211 raft_consensus.cc:504] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:13.117038 20124 raft_consensus.cc:504] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:13.491840 16501 raft_consensus.cc:1226] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6. Ops: []
W20260810 18:52:13.399250 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 392740us
W20260810 18:52:13.402273 17726 leader_election.cc:334] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.224s (SENT)
I20260810 18:52:13.492785 17726 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:13.492975 20124 leader_election.cc:288] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:13.493551 20124 consensus_queue.cc:1035] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:52:13.494261 19786 raft_consensus.cc:2726] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:13.399250 20172 leader_election.cc:288] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:13.104461 19982 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:13.495188 20172 raft_consensus.cc:482] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:13.484622 20231 log.cc:924] Time spent T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.328s user 0.000s sys 0.001s
W20260810 18:52:13.485538 20237 log.cc:924] Time spent T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.342s user 0.001s sys 0.000s
I20260810 18:52:13.495730 20172 raft_consensus.cc:504] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:13.498045 20172 leader_election.cc:288] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:13.498385 20211 leader_election.cc:288] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:13.165256 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 187875us
W20260810 18:52:13.493757 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 91548.3us
I20260810 18:52:13.498764 16624 raft_consensus.cc:3032] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:52:13.499207 16624 raft_consensus.cc:729] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:13.483387 20238 raft_consensus.cc:504] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:13.499962 16624 consensus_queue.cc:260] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:13.501121 16624 raft_consensus.cc:3037] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:13.475927 20257 raft_consensus.cc:482] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:13.505383 20257 raft_consensus.cc:504] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
W20260810 18:52:13.491770 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 89608.3us
I20260810 18:52:13.508975 16624 raft_consensus.cc:1261] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260810 18:52:13.528254 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:13.529160 16749 raft_consensus.cc:3037] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:13.540730 16749 raft_consensus.cc:2454] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
I20260810 18:52:13.424448 20047 raft_consensus.cc:504] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:13.546778 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:13.547609 16506 raft_consensus.cc:2454] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:52:13.548604 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:13.549136 16573 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:13.549726 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:13.552932 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:13.552999 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:13.553753 16749 raft_consensus.cc:3037] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 6
I20260810 18:52:13.554737 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:13.555490 16746 raft_consensus.cc:3037] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:13.498787 20172 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:13.582958 20233 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:13.583659 20233 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:13.584137 20233 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:13.586112 20233 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:13.588232 16749 raft_consensus.cc:2454] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
I20260810 18:52:13.589079 20172 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:13.597711 20257 leader_election.cc:288] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:13.595688 19561 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:13.598692 19561 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:13.600183 20047 leader_election.cc:288] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:13.601076 19561 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:13.591238 20234 raft_consensus.cc:2781] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:13.601982 20234 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:13.602463 20234 raft_consensus.cc:3037] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:13.609802 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:13.610690 16752 raft_consensus.cc:3037] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:13.611797 16506 raft_consensus.cc:1226] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:13.613581 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:13.614600 16746 raft_consensus.cc:2454] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:13.597745 20172 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:13.605207 20238 leader_election.cc:288] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:13.619910 20190 leader_election.cc:288] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:13.643672 20039 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:13.664784 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:13.619316 20256 consensus_queue.cc:1046] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20260810 18:52:13.677079 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:13.679199 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:13.661253 20190 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:13.686546 16752 raft_consensus.cc:2454] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
I20260810 18:52:13.696130 16742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1630) took 3139 ms. Trace:
W20260810 18:52:13.723912 20268 log.cc:924] Time spent T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20260810 18:52:13.693470 19099 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
W20260810 18:52:13.725495 20189 log.cc:924] Time spent T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 2.279s user 0.003s sys 0.017s
I20260810 18:52:13.672771 20039 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:13.727360 20263 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:13.728333 20039 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:13.728008 20263 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:13.729048 17781 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55586 (request call id 325) took 3722 ms (client timeout 1657 ms). Trace:
I20260810 18:52:13.729557 20039 raft_consensus.cc:482] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:13.727361 19244 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:13.725167 19099 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:13.731768 17783 raft_consensus.cc:1226] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 5 LEADER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:13.697784 20211 raft_consensus.cc:482] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:13.696583 16742 rpcz_store.cc:276] 0810 18:52:10.545153 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:10.545418 (+ 265us) service_pool.cc:227] Handling call
0810 18:52:10.618870 (+ 73452us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:10.619206 (+ 336us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:10.619229 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:13.060130 (+2440901us) spinlock_profiling.cc:243] Waited 16.4 ms on lock 0x7b2c00159598. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae6adde11 00007f1ae6aed442 00007f1ae8135c11 00007f1ae80fa58a 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772
0810 18:52:13.060563 (+ 433us) log.cc:842] Serialized 26 byte log entry
0810 18:52:13.484585 (+424022us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:13.484626 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:13.484699 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:13.680922 (+196223us) raft_consensus.cc:1671] finished
0810 18:52:13.680948 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:13.684800 (+ 3852us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":568,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":108,"spinlock_wait_cycles":37647616,"thread_start_us":339495,"threads_started":1,"wal-append.queue_time_us":489671}
I20260810 18:52:13.735898 16450 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:13.738758 16749 raft_consensus.cc:1226] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260810 18:52:13.740226 16450 leader_election.cc:302] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:13.743656 16500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1444) took 2771 ms. Trace:
I20260810 18:52:13.744814 16500 rpcz_store.cc:276] 0810 18:52:10.972510 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:10.972656 (+ 146us) service_pool.cc:227] Handling call
0810 18:52:10.973058 (+ 402us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:10.973432 (+ 374us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:10.973463 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:10.973924 (+ 461us) log.cc:842] Serialized 26 byte log entry
0810 18:52:11.171194 (+197270us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:11.171232 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:11.171299 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:13.724837 (+2553538us) spinlock_profiling.cc:243] Waited 8.48 ms on lock 0x7b44000232cc. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae50e4a95 00007f1ae50e54b2 00007f1ae7d8d511 00007f1ae7d88e1a 00007f1ae7d7cbee 00007f1ae7d7bf36 00007f1ae7d7bb58 00007f1ae7d85551 00007f1ae7d85509 00007f1ae7d85499
0810 18:52:13.743343 (+ 18506us) raft_consensus.cc:1671] finished
0810 18:52:13.743370 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:13.743588 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":89,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89,"spinlock_wait_cycles":19498240,"thread_start_us":196873,"threads_started":1,"wal-append.queue_time_us":471787}
I20260810 18:52:13.742331 19244 raft_consensus.cc:2781] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election won for term 7
I20260810 18:52:13.741719 20211 raft_consensus.cc:504] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:13.729554 17781 rpcz_store.cc:269] 0810 18:52:10.006419 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:10.028467 (+ 22048us) service_pool.cc:227] Handling call
0810 18:52:13.728743 (+3700276us) spinlock_profiling.cc:243] Waited 765 ms on lock 0x7b54005e0454. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:13.728917 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1758419200}
I20260810 18:52:13.756907 20211 leader_election.cc:288] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:13.738755 20067 leader_election.cc:288] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:13.764693 16701 leader_election.cc:334] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.228s (SENT)
I20260810 18:52:13.703927 20190 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:13.730213 20039 raft_consensus.cc:504] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:13.772306 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 45886.6us
I20260810 18:52:13.746822 20282 raft_consensus.cc:482] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:13.753510 19244 raft_consensus.cc:686] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:13.763720 20289 raft_consensus.cc:2781] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:13.808808 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:13.779961 20263 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:13.791601 20190 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:13.792583 16495 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56468 (request call id 193) took 3718 ms (client timeout 2189 ms). Trace:
I20260810 18:52:13.793371 16621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1575) took 4275 ms. Trace:
I20260810 18:52:13.795100 20039 leader_election.cc:288] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:13.795353 16507 raft_consensus.cc:1226] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:13.823009 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:13.796648 19099 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:13.797545 19786 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:13.797650 19244 consensus_queue.cc:237] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:13.848268 16636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1582) took 4232 ms. Trace:
I20260810 18:52:13.798745 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:13.791929 19858 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:13.800170 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:13.801111 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:13.801224 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:13.801751 20110 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:13.804158 16632 raft_consensus.cc:3032] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:13.867792 16632 raft_consensus.cc:729] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:13.804827 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:13.805931 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:52:13.810316 16495 rpcz_store.cc:269] 0810 18:52:10.074201 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:10.074321 (+ 120us) service_pool.cc:227] Handling call
0810 18:52:13.792312 (+3717991us) spinlock_profiling.cc:243] Waited 574 ms on lock 0x7b54001cbbd4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:13.792455 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1320531712}
I20260810 18:52:13.870208 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:13.870985 16495 raft_consensus.cc:2379] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 6: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:13.849954 17780 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55570 (request call id 422) took 4727 ms. Trace:
I20260810 18:52:13.825342 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:13.884163 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:13.831529 16507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1446) took 2857 ms. Trace:
I20260810 18:52:13.845675 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:13.885516 16507 rpcz_store.cc:276] 0810 18:52:10.973857 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:10.974041 (+ 184us) service_pool.cc:227] Handling call
0810 18:52:13.795165 (+2821124us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:13.822338 (+ 27173us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:13.831307 (+ 8969us) spinlock_profiling.cc:243] Waited 2.82 s on lock 0x7b54001cbbd0. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:13.831463 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6487208192}
I20260810 18:52:13.883847 16573 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:13.848624 20289 raft_consensus.cc:686] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 6 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:13.857105 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:13.848891 16636 rpcz_store.cc:276] 0810 18:52:09.592657 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:09.592783 (+ 126us) service_pool.cc:227] Handling call
0810 18:52:09.618659 (+ 25876us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:09.763774 (+145115us) raft_consensus.cc:1520] Early marking committed up to index 2
0810 18:52:09.764182 (+ 408us) log.cc:842] Serialized 14 byte log entry
0810 18:52:10.006897 (+242715us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:10.007470 (+ 573us) log.cc:842] Serialized 26 byte log entry
0810 18:52:10.007619 (+ 149us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:10.007662 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:13.785382 (+3777720us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b54001ec154. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:13.785450 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:13.785492 (+ 42us) raft_consensus.cc:1671] finished
0810 18:52:13.824542 (+ 39050us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:13.824686 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25622,"prepare.queue_time_us":242,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":26946,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":26188,"spinlock_wait_cycles":332587776,"thread_start_us":216559,"threads_started":1,"wal-append.queue_time_us":319041,"wal-append.run_cpu_time_us":2232,"wal-append.run_wall_time_us":1006826}
I20260810 18:52:09.874223 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:52:13.871954 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:13.890534 16495 raft_consensus.cc:2362] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 4. Re-sending same reply.
I20260810 18:52:13.815181 16749 raft_consensus.cc:1261] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260810 18:52:13.883267 17781 raft_consensus.cc:2379] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 6: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:13.878060 19391 raft_consensus.cc:482] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:13.878628 20073 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:13.851585 19786 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:13.879594 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:13.859917 19858 raft_consensus.cc:3037] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:13.881196 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:52:13.848949 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 95719.6us
I20260810 18:52:13.886796 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:13.800529 16324 catalog_manager.cc:4841] TS 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): tablet 7ef7c65533724e678b75830002906e2c (table test-workload [id=5aa4952cbee8447ca410249e65af5af6]) successfully deleted
I20260810 18:52:13.787660 20282 raft_consensus.cc:504] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:13.888235 16573 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:13.821918 19162 consensus_queue.cc:1035] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20260810 18:52:13.886740 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:13.923439 16507 raft_consensus.cc:2454] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 4.
W20260810 18:52:13.891767 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 60417.6us
I20260810 18:52:13.868762 16632 consensus_queue.cc:260] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:13.867816 17770 raft_consensus.cc:3037] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:13.892962 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:13.893595 17787 raft_consensus.cc:1261] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260810 18:52:13.894019 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:13.891989 19940 raft_consensus.cc:2726] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:13.901633 16500 raft_consensus.cc:1226] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:13.917904 16605 tablet_service.cc:1514] Processing DeleteTablet for tablet cdfc652dd163415b94d3f0bc3931a07f with delete_type TABLET_DATA_TOMBSTONED (TS 0397d760589d40d3b92ef141cdfc2dbb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53868
I20260810 18:52:13.918524 19786 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:13.918828 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:13.919631 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:13.920454 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:13.924983 16632 raft_consensus.cc:3037] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:13.931839 16740 raft_consensus.cc:2354] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 6. Current term is 7.
I20260810 18:52:13.932113 16750 raft_consensus.cc:2379] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 3: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:13.926239 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:13.932633 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:13.933496 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:13.928670 16573 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:13.933488 16740 raft_consensus.cc:2354] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6.
I20260810 18:52:13.812536 16621 rpcz_store.cc:276] 0810 18:52:09.518346 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:13.786462 (+4268116us) service_pool.cc:227] Handling call
0810 18:52:13.792781 (+ 6319us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:13.792983 (+ 202us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:13.793011 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:13.793058 (+ 47us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:13.793099 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:13.793170 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:13.793307 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:13.932682 16636 raft_consensus.cc:3037] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
W20260810 18:52:13.933113 16324 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 132792us
I20260810 18:52:13.939558 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:13.940398 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:13.933625 17775 raft_consensus.cc:2396] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for term 6 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260810 18:52:13.941550 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:13.942824 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:13.946401 16749 raft_consensus.cc:2396] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260810 18:52:13.943648 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:13.927554 20256 raft_consensus.cc:2726] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:13.900213 20234 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:13.956717 20234 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:13.958350 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:13.928228 17771 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:13.959753 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:13.961947 16749 raft_consensus.cc:2354] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6.
I20260810 18:52:13.904594 20073 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:13.902077 19391 raft_consensus.cc:504] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:13.941196 19940 raft_consensus.cc:2726] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:52:13.967648 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:13.968248 16507 raft_consensus.cc:3037] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:13.964905 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:13.965603 16450 leader_election.cc:302] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:13.966292 16742 raft_consensus.cc:1226] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:13.970227 17787 raft_consensus.cc:2396] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for term 6 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260810 18:52:13.941388 16502 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52338 (request call id 1410) took 4059 ms (client timeout 1809 ms). Trace:
I20260810 18:52:13.974135 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
W20260810 18:52:13.971993 17726 leader_election.cc:334] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.712s (SENT)
I20260810 18:52:13.976259 17726 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:13.976938 16573 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:13.980175 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 55511.6us
I20260810 18:52:13.982429 16450 leader_election.cc:302] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:13.984050 16636 raft_consensus.cc:2454] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:13.985819 16507 raft_consensus.cc:2454] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:13.986160 16632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1454) took 1993 ms. Trace:
I20260810 18:52:13.986388 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:14.031838 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:13.988219 17724 connection.cc:232] RPC call timeout handler was delayed by -1.1924431880470365s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2195s for a response
I20260810 18:52:13.962966 20289 consensus_queue.cc:237] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:13.990263 16573 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:13.986496 16632 rpcz_store.cc:276] 0810 18:52:11.992947 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:13.803670 (+1810723us) service_pool.cc:227] Handling call
0810 18:52:13.804029 (+ 359us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:13.985832 (+181803us) raft_consensus.cc:1520] Early marking committed up to index 5
0810 18:52:13.985863 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:13.985902 (+ 39us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:13.985942 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:13.986005 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:13.986107 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":218}
I20260810 18:52:13.969025 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:52:14.039857 17724 connection.cc:232] RPC call timeout handler was delayed by -1.0872329270932823s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1533s for a response
W20260810 18:52:14.040659 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 50479.9us
I20260810 18:52:14.042098 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:13.996160 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:14.043226 16752 raft_consensus.cc:2396] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260810 18:52:13.987995 16448 leader_election.cc:302] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:52:14.044374 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 56466.4us
I20260810 18:52:13.989413 20297 raft_consensus.cc:2726] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260810 18:52:13.972826 16502 rpcz_store.cc:269] 0810 18:52:09.881060 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:09.881690 (+ 630us) service_pool.cc:227] Handling call
0810 18:52:13.879400 (+3997710us) spinlock_profiling.cc:243] Waited 1.52 ms on lock 0x7b5800001898. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed03c1b0 00007f1aed03ba94 00007f1ae6723c04 00007f1ae6723b92
0810 18:52:13.937032 (+ 57632us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3486080}
I20260810 18:52:14.046221 16636 raft_consensus.cc:3037] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:13.981307 19940 raft_consensus.cc:2726] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:13.979353 20295 raft_consensus.cc:2781] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:52:13.882556 17780 rpcz_store.cc:276] 0810 18:52:09.094940 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:09.095313 (+ 373us) service_pool.cc:227] Handling call
0810 18:52:12.994697 (+3899384us) spinlock_profiling.cc:243] Waited 30.9 ms on lock 0x7b5800005498. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed0502b0 00007f1aed03b7f9 00007f1aed03b2a2 00007f1ae6721684
0810 18:52:13.012328 (+ 17631us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:13.064778 (+ 52450us) raft_consensus.cc:1520] Early marking committed up to index 5
0810 18:52:13.556571 (+491793us) log.cc:842] Serialized 14 byte log entry
0810 18:52:13.678240 (+121669us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:13.745279 (+ 67039us) log.cc:842] Serialized 26 byte log entry
0810 18:52:13.754228 (+ 8949us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:13.766023 (+ 11795us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:13.768443 (+ 2420us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:13.769135 (+ 692us) raft_consensus.cc:1671] finished
0810 18:52:13.786535 (+ 17400us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:13.821722 (+ 35187us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":474492,"prepare.run_cpu_time_us":290,"prepare.run_wall_time_us":1102,"raft.queue_time_us":18504,"raft.run_cpu_time_us":319,"raft.run_wall_time_us":20276,"spinlock_wait_cycles":70969088,"thread_start_us":423226,"threads_started":2,"wal-append.queue_time_us":153598}
I20260810 18:52:13.991096 20296 raft_consensus.cc:2781] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:14.065753 20296 raft_consensus.cc:686] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 6 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:14.066120 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:14.001932 20308 tablet_replica.cc:330] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb: stopping tablet replica
I20260810 18:52:13.993479 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:14.050760 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 55408.4us
I20260810 18:52:14.052297 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.069628 16742 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:14.053416 16636 raft_consensus.cc:2454] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:14.050240 20301 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:14.040532 20256 raft_consensus.cc:2781] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:14.055833 20282 leader_election.cc:288] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:14.062839 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:14.063483 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:14.068702 16745 raft_consensus.cc:1226] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:14.071349 16448 leader_election.cc:302] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:14.071833 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:14.084163 16636 raft_consensus.cc:2354] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for earlier term 4. Current term is 5.
I20260810 18:52:14.084883 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:14.085121 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:14.086057 16636 raft_consensus.cc:2354] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for earlier term 5. Current term is 6.
I20260810 18:52:14.087404 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:14.086903 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.089048 17724 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:14.066844 17781 raft_consensus.cc:3037] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:14.051576 20249 raft_consensus.cc:2781] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:14.090123 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:14.091732 16746 raft_consensus.cc:1261] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260810 18:52:14.091871 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:14.092895 16575 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60066 dropped due to backpressure. The service queue is full; it has 50 items.
I20260810 18:52:14.094209 16636 raft_consensus.cc:2454] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:14.085522 16752 raft_consensus.cc:2454] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:14.098410 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:14.097975 20316 raft_consensus.cc:2781] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:14.099408 16636 raft_consensus.cc:2354] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for earlier term 4. Current term is 5.
I20260810 18:52:14.100440 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.101235 16636 raft_consensus.cc:2454] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 3.
I20260810 18:52:14.102507 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:14.047382 19786 consensus_queue.cc:1035] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20260810 18:52:14.105839 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:14.081231 20295 raft_consensus.cc:686] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:14.109515 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:14.110179 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:14.110620 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:14.113766 16746 raft_consensus.cc:3037] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:52:14.086980 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:14.102447 20256 raft_consensus.cc:686] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:14.123299 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:14.066774 20296 consensus_queue.cc:237] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:14.116815 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:14.125597 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:14.133358 16749 raft_consensus.cc:1226] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:14.132258 16746 raft_consensus.cc:2454] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:52:14.124872 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:14.135980 20073 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:14.090173 19940 consensus_queue.cc:1046] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.025s
I20260810 18:52:14.113989 20110 raft_consensus.cc:2726] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:14.094440 19244 consensus_queue.cc:1046] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260810 18:52:14.108291 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:14.140115 16746 raft_consensus.cc:1226] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260810 18:52:14.138278 16749 raft_consensus.cc:1226] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260810 18:52:14.109078 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:14.090757 20301 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:14.146209 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:14.146675 19718 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:14.148355 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:14.148834 16631 raft_consensus.cc:1261] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260810 18:52:14.150166 16633 raft_consensus.cc:3032] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:14.136948 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:14.152869 19899 leader_election.cc:288] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:14.155422 16743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1650) took 2509 ms. Trace:
I20260810 18:52:14.154870 19858 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:14.158082 20202 log.cc:924] Time spent T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 2.359s user 0.015s sys 0.005s
I20260810 18:52:14.162061 17770 raft_consensus.cc:2454] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:52:14.145130 20295 consensus_queue.cc:237] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:14.107611 16575 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60066 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 157)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 167)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 170)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 1627)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 176)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 154)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 1639)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 166)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 163)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 181)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 164)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 178)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 165)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 177)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 172)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 171)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 179)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 180)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 175)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 174)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 168)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 173)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 1634)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1614)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1615)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1619)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1620)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1621)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1622)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1623)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1624)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1625)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1628)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1629)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1632)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1633)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1635)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1636)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1640)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1641)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60066 (request call id 155)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60066 (request call id 156)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60066 (request call id 158)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1642)
I20260810 18:52:14.191002 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:14.289911 16745 raft_consensus.cc:1261] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260810 18:52:14.322209 16633 raft_consensus.cc:729] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:14.322316 16450 leader_election.cc:302] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:14.300289 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 207666us
I20260810 18:52:14.142143 19391 leader_election.cc:288] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:14.324581 16750 raft_consensus.cc:3037] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 5
I20260810 18:52:14.315264 16746 raft_consensus.cc:1226] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6. Ops: [3.2-5.3]
W20260810 18:52:14.173696 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 102428us
I20260810 18:52:14.325328 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:14.198038 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:14.326452 16749 raft_consensus.cc:3032] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:52:14.315843 20249 raft_consensus.cc:686] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:14.286486 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:14.284811 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:14.266558 20263 consensus_queue.cc:1035] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.080s
I20260810 18:52:14.442036 16747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1644) took 2869 ms. Trace:
I20260810 18:52:14.447957 16628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1589) took 3475 ms. Trace:
I20260810 18:52:14.478322 16628 rpcz_store.cc:276] 0810 18:52:10.972073 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:11.012164 (+ 40091us) service_pool.cc:227] Handling call
0810 18:52:11.012545 (+ 381us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:11.224698 (+212153us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:11.224726 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:11.224764 (+ 38us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:52:11.224791 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:14.447648 (+3222857us) spinlock_profiling.cc:243] Waited 212 ms on lock 0x7b54002a5154. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:14.447718 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:14.447871 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":487460224}
I20260810 18:52:14.445943 16504 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1420) took 1811 ms. Trace:
I20260810 18:52:14.448849 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:14.338304 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:14.482190 16504 rpcz_store.cc:276] 0810 18:52:12.634365 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:14.445127 (+1810762us) service_pool.cc:227] Handling call
0810 18:52:14.445476 (+ 349us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:14.445645 (+ 169us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:14.445669 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:14.445706 (+ 37us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:14.445739 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:14.445786 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:14.445887 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 18:52:14.441651 20046 log.cc:924] Time spent T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 4.927s user 0.044s sys 0.038s
I20260810 18:52:14.331658 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:14.478143 16747 rpcz_store.cc:276] 0810 18:52:11.572820 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:11.572930 (+ 110us) service_pool.cc:227] Handling call
0810 18:52:11.573265 (+ 335us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:11.751586 (+178321us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:11.751616 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:11.751656 (+ 40us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:11.751700 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:14.441794 (+2690094us) spinlock_profiling.cc:243] Waited 178 ms on lock 0x7b54003be554. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:14.441839 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:14.441970 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":409591168}
I20260810 18:52:14.364444 16505 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1449) took 3388 ms. Trace:
I20260810 18:52:14.452039 16749 raft_consensus.cc:729] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
W20260810 18:52:14.452281 20327 log.cc:924] Time spent T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.163s user 0.000s sys 0.001s
I20260810 18:52:14.452587 16752 raft_consensus.cc:2454] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:52:14.507932 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:14.508287 16750 raft_consensus.cc:1261] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260810 18:52:14.508600 16752 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
W20260810 18:52:14.454524 16701 leader_election.cc:334] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.887s (SENT)
I20260810 18:52:14.509684 16701 leader_election.cc:302] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:14.510867 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 56389.6us
W20260810 18:52:14.511675 16701 leader_election.cc:334] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.981s (SENT)
I20260810 18:52:14.512140 16701 leader_election.cc:302] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:14.513895 16701 leader_election.cc:334] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.580s (SENT)
I20260810 18:52:14.514329 16701 leader_election.cc:302] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:14.456954 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:14.467243 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:14.515918 16622 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1459) took 2295 ms (client timeout 2580 ms). Trace:
W20260810 18:52:14.516390 16622 rpcz_store.cc:269] 0810 18:52:12.220117 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:14.466799 (+2246682us) service_pool.cc:227] Handling call
0810 18:52:14.515836 (+ 49037us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:14.469903 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.470337 16496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1421) took 1835 ms. Trace:
I20260810 18:52:14.188062 19940 consensus_queue.cc:1225] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer d304ac47fdf846de81337057a4ef7bea log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260810 18:52:14.341259 17787 raft_consensus.cc:1226] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:14.524945 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:14.323781 16743 rpcz_store.cc:276] 0810 18:52:11.646287 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:11.646455 (+ 168us) service_pool.cc:227] Handling call
0810 18:52:14.154727 (+2508272us) spinlock_profiling.cc:243] Waited 2.58 ms on lock 0x7b5800270f98. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed0502b0 00007f1aed03b7f9 00007f1aed03b2a2 00007f1ae6721684
0810 18:52:14.154920 (+ 193us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:14.155116 (+ 196us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:14.155141 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:14.155178 (+ 37us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:14.155209 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:14.155260 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:14.155369 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5932288}
I20260810 18:52:14.341270 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:14.363300 17786 raft_consensus.cc:2454] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:14.450615 16575 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60066 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 170)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 1627)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 176)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 154)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 1639)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 166)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 163)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 181)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 164)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 178)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 165)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 182)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 177)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 172)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 171)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 179)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 180)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 190)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 183)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 186)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 188)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 175)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 187)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 174)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 185)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 168)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 173)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 184)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 1634)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1614)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1615)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1619)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1620)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1621)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1622)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1623)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1624)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1625)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1628)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1629)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1632)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1633)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1635)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1636)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1640)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1641)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60066 (request call id 155)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60066 (request call id 156)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60066 (request call id 158)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1642)
W20260810 18:52:14.500506 16344 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37022 (request call id 213) took 13333 ms (client timeout 15000 ms). Trace:
I20260810 18:52:14.527840 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:14.345221 16631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1452) took 2378 ms. Trace:
I20260810 18:52:14.450990 20249 consensus_queue.cc:237] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:14.501857 16745 raft_consensus.cc:2396] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260810 18:52:14.492364 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 150377us
I20260810 18:52:14.519965 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:14.528806 16631 rpcz_store.cc:276] 0810 18:52:11.966983 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:14.148085 (+2181102us) service_pool.cc:227] Handling call
0810 18:52:14.148446 (+ 361us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:14.323437 (+174991us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:14.345148 (+ 21711us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:14.533385 16575 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53870 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 1639)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 166)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 163)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 181)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 164)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 178)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 165)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 182)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 177)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 172)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 171)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 179)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 180)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 190)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 183)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 186)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1463)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 188)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 175)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1464)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 187)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1465)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 174)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 185)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 168)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 173)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 184)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1466)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 1634)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1614)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1615)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1619)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1620)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1621)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1622)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1623)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1624)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1625)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1628)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1629)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1632)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1633)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1635)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1636)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1640)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1641)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60066 (request call id 155)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60066 (request call id 156)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60066 (request call id 158)
I20260810 18:52:14.521049 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:14.523598 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.330186 16633 consensus_queue.cc:260] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:14.538589 16636 raft_consensus.cc:2396] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260810 18:52:14.538441 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:14.538803 16633 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:14.539235 16743 raft_consensus.cc:2379] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 3: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:14.522365 16496 rpcz_store.cc:276] 0810 18:52:12.634740 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:12.637820 (+ 3080us) service_pool.cc:227] Handling call
0810 18:52:12.723538 (+ 85718us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:12.823481 (+ 99943us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:14.470095 (+1646614us) spinlock_profiling.cc:243] Waited 89.7 ms on lock 0x7b54001cc0d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:14.470274 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":206272768}
W20260810 18:52:14.530249 16695 connection.cc:232] RPC call timeout handler was delayed by -2.4160738750360906s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23750000000000002s for a response
I20260810 18:52:14.501786 16505 rpcz_store.cc:276] 0810 18:52:10.976183 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:10.980569 (+ 4386us) service_pool.cc:227] Handling call
0810 18:52:11.007711 (+ 27142us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:11.008115 (+ 404us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:11.008153 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:11.079827 (+ 71674us) log.cc:842] Serialized 26 byte log entry
0810 18:52:11.576218 (+496391us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:52:11.576288 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:11.576377 (+ 89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:14.157430 (+2581053us) spinlock_profiling.cc:243] Waited 7.92 ms on lock 0x7b44000232cc. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae50e4a95 00007f1ae50e54b2 00007f1ae7d8d511 00007f1ae7d88e1a 00007f1ae7d7cbee 00007f1ae7d7bf36 00007f1ae7d7bb58 00007f1ae7d85551 00007f1ae7d85509 00007f1ae7d85499
0810 18:52:14.346570 (+189140us) raft_consensus.cc:1671] finished
0810 18:52:14.346598 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:14.364380 (+ 17782us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":12867,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":13157,"spinlock_wait_cycles":18217344,"thread_start_us":406021,"threads_started":1,"wal-append.queue_time_us":662414}
I20260810 18:52:14.500286 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:14.285913 20256 consensus_queue.cc:237] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:14.546458 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:14.547298 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.547693 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:14.547824 17783 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55570 (request call id 421) took 5672 ms. Trace:
I20260810 18:52:14.341602 20318 raft_consensus.cc:2781] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:14.537436 16749 consensus_queue.cc:260] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:14.556025 16749 raft_consensus.cc:3037] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
W20260810 18:52:14.544984 16695 connection.cc:232] RPC call timeout handler was delayed by -1.3127544748131186s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16570000000000001s for a response
W20260810 18:52:14.556905 16695 connection.cc:232] RPC call timeout handler was delayed by -1.1189239518716931s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1636s for a response
I20260810 18:52:14.520191 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:14.548372 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.543506 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:14.557927 16636 raft_consensus.cc:2454] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 5.
I20260810 18:52:14.427009 17781 raft_consensus.cc:2454] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
W20260810 18:52:14.557251 16695 connection.cc:232] RPC call timeout handler was delayed by -1.0750561780296266s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1686s for a response
I20260810 18:52:14.557632 16622 raft_consensus.cc:2354] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for earlier term 4. Current term is 5.
I20260810 18:52:14.561256 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.561483 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:14.548307 17783 rpcz_store.cc:276] 0810 18:52:08.875779 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:08.875930 (+ 151us) service_pool.cc:227] Handling call
0810 18:52:10.940119 (+2064189us) spinlock_profiling.cc:243] Waited 25.3 ms on lock 0x7b5800005498. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed0502b0 00007f1aed03b7f9 00007f1aed03b2a2 00007f1ae6721684
0810 18:52:13.731496 (+2791377us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:13.736090 (+ 4594us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:14.547580 (+811490us) spinlock_profiling.cc:243] Waited 2.79 s on lock 0x7b54005e0450. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:14.547751 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6477204608}
I20260810 18:52:14.540619 16575 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60066 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 166)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 163)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 181)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 164)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 178)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 165)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 182)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 177)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 172)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 171)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 179)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 180)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 190)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 183)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 186)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1463)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 188)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 175)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1464)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 187)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1467)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1468)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1465)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 174)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 185)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 168)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1469)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 173)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 184)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1466)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 1634)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1614)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1615)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1619)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1620)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1621)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1622)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1623)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1624)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1625)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1628)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1629)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1632)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1633)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1635)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1636)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1640)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1641)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60066 (request call id 155)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60066 (request call id 156)
W20260810 18:52:14.560588 16695 connection.cc:232] RPC call timeout handler was delayed by -1.0742632299661636s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1686s for a response
I20260810 18:52:14.525328 20067 raft_consensus.cc:2726] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:52:14.559320 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:14.525178 20037 raft_consensus.cc:2726] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:14.559001 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:14.568157 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 479211us
I20260810 18:52:14.566306 16575 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60066 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 177)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 172)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1471)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1474)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1470)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 171)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 179)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1477)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 180)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 190)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 183)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1472)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 186)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1463)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1473)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1475)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 188)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 175)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1464)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 187)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1467)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1468)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1465)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1476)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 174)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 185)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 168)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1469)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 173)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 184)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1466)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 1634)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1614)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1615)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1619)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1620)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1621)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1622)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1623)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1624)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1625)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1628)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1629)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1632)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1633)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1635)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1636)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1640)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1641)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60066 (request call id 155)
I20260810 18:52:14.568683 16636 raft_consensus.cc:2454] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 3.
I20260810 18:52:14.569487 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.564324 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.569494 16575 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60066 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 172)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1471)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1474)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1478)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1470)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 171)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 179)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1477)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 180)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 190)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 183)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1472)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 186)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1463)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1473)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1475)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 188)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 175)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1464)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 187)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1467)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1468)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1465)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1476)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 174)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 185)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 168)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1469)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 173)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 184)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1466)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 1634)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1614)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1615)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1619)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1620)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1621)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1622)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1623)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1624)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1625)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1628)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1629)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1632)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1633)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1635)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1636)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1640)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1641)
I20260810 18:52:14.573150 16631 raft_consensus.cc:2396] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260810 18:52:14.570103 16636 raft_consensus.cc:2454] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
I20260810 18:52:14.543172 19929 raft_consensus.cc:2726] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:14.573943 16575 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53870 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1474)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1478)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1470)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 171)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 179)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1477)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 180)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 190)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 183)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1472)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 186)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1463)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1473)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1475)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 188)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 175)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1464)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 187)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1467)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1468)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1465)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1476)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 174)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 185)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 168)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1469)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 173)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 184)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1479)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1466)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1480)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 1634)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1614)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1615)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1619)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1620)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1621)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1622)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1623)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1624)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1625)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1628)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1629)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1632)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1633)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1635)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1636)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1640)
I20260810 18:52:14.537292 16621 raft_consensus.cc:2454] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 3.
I20260810 18:52:14.577136 16746 raft_consensus.cc:1261] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260810 18:52:14.586956 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:14.591590 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 54048.5us
I20260810 18:52:14.554793 20282 raft_consensus.cc:482] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:14.581673 16749 raft_consensus.cc:2454] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:14.589644 16695 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:14.588533 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:14.595798 16636 raft_consensus.cc:3037] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:14.555435 20318 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:14.587980 17724 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:14.560581 17770 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:14.589759 16575 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53870 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1484)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1478)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1470)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 171)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 179)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1477)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 180)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 190)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 183)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1472)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 186)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1481)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1463)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1473)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1475)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 188)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 175)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1464)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1483)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 187)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1467)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1468)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1465)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1476)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 174)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 185)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 168)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1469)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 173)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 184)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1479)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1482)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1466)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1480)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 1634)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1614)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1615)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1619)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1620)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1621)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1622)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1623)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1624)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1625)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1628)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1629)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1632)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1633)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1635)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1636)
I20260810 18:52:14.600514 17724 leader_election.cc:302] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:14.581226 16633 raft_consensus.cc:1261] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260810 18:52:14.601164 16633 pending_rounds.cc:76] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb: Aborting all ops after (but not including) 3
I20260810 18:52:14.612063 20035 raft_consensus.cc:2781] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260810 18:52:14.528251 16344 rpcz_store.cc:269] 0810 18:52:01.166802 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:01.166967 (+ 165us) service_pool.cc:227] Handling call
0810 18:52:14.431045 (+13264078us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 18:52:05.771692 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 18:52:05.772722 (+ 1030us) write_op.cc:432] Acquiring schema lock in shared mode
0810 18:52:05.772780 (+ 58us) write_op.cc:435] Acquired schema lock
0810 18:52:05.772821 (+ 41us) tablet.cc:660] Decoding operations
0810 18:52:05.775313 (+ 2492us) write_op.cc:620] Acquiring the partition lock for write op
0810 18:52:05.775449 (+ 136us) write_op.cc:641] Partition lock acquired for write op
0810 18:52:05.775508 (+ 59us) tablet.cc:683] Acquiring locks for 9 operations
0810 18:52:05.777080 (+ 1572us) tablet.cc:699] Row locks acquired
0810 18:52:05.777119 (+ 39us) write_op.cc:260] PREPARE: finished
0810 18:52:05.777389 (+ 270us) write_op.cc:270] Start()
0810 18:52:05.777585 (+ 196us) write_op.cc:276] Timestamp: P: 1786387925777294 usec, L: 0
0810 18:52:05.777633 (+ 48us) op_driver.cc:331] REPLICATION: starting
0810 18:52:05.779265 (+ 1632us) log.cc:842] Serialized 3641 byte log entry
0810 18:52:05.902632 (+123367us) spinlock_profiling.cc:243] Waited 18.1 ms on lock 0x7b44000232cc. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae50e4a95 00007f1ae50e54b2 00007f1ae7d8d511 00007f1ae7d88e1a 00007f1ae51b45fe 00007f1aedfeba39 00007f1ae51d18bb 00007f1ae51cfcbf 00007f1ae51d1e22 00007f1ae7d7b5bb
0810 18:52:06.186512 (+283880us) op_driver.cc:444] REPLICATION: finished
0810 18:52:06.686863 (+500351us) write_op.cc:301] APPLY: starting
0810 18:52:06.698247 (+ 11384us) tablet.cc:1365] starting BulkCheckPresence
0810 18:52:06.804656 (+106409us) tablet.cc:1368] finished BulkCheckPresence
0810 18:52:06.804688 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 18:52:06.812976 (+ 8288us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 18:52:06.813306 (+ 330us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0810 18:52:06.813366 (+ 60us) write_op.cc:312] APPLY: finished
0810 18:52:06.813811 (+ 445us) log.cc:842] Serialized 71 byte log entry
0810 18:52:06.814000 (+ 189us) write_op.cc:489] Releasing partition, row and schema locks
0810 18:52:06.814143 (+ 143us) write_op.cc:454] Released schema lock
0810 18:52:06.814202 (+ 59us) write_op.cc:341] FINISH: Updating metrics
0810 18:52:06.814555 (+ 353us) write_op.cc:489] Releasing partition, row and schema locks
0810 18:52:06.814595 (+ 40us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":206825,"reported_tablets":19,"child_traces":[["op",{"apply.queue_time_us":472546,"apply.run_cpu_time_us":4590,"apply.run_wall_time_us":136114,"num_ops":9,"prepare.queue_time_us":505175,"prepare.run_cpu_time_us":22682,"prepare.run_wall_time_us":219258,"raft.queue_time_us":147408,"raft.run_cpu_time_us":11188,"raft.run_wall_time_us":137817,"replication_time_us":399675,"spinlock_wait_cycles":41681280,"thread_start_us":898769,"threads_started":4,"wal-append.queue_time_us":219047,"wal-append.run_cpu_time_us":14127,"wal-append.run_wall_time_us":1824631}]]}
I20260810 18:52:14.612620 20035 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:14.613049 20035 raft_consensus.cc:3037] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:14.601433 16633 pending_rounds.cc:97] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260810 18:52:14.615294 16633 raft_consensus.cc:2866] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:52:14.612154 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.615985 16633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1448) took 2818 ms. Trace:
I20260810 18:52:14.621722 16633 rpcz_store.cc:276] 0810 18:52:11.845763 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:14.149620 (+2303857us) service_pool.cc:227] Handling call
0810 18:52:14.150007 (+ 387us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:14.615763 (+465756us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:14.615933 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:14.622668 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.615875 16695 leader_election.cc:302] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:14.624486 16633 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:14.625294 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.642406 16633 raft_consensus.cc:2379] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 4: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:14.643784 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.644690 16633 raft_consensus.cc:2454] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 5.
I20260810 18:52:14.645865 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.572543 19718 consensus_queue.cc:1046] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260810 18:52:14.597908 17771 raft_consensus.cc:1226] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:14.653661 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:14.609103 20067 raft_consensus.cc:2726] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:14.659014 16749 raft_consensus.cc:3037] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:14.660041 16636 raft_consensus.cc:2454] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 4.
I20260810 18:52:14.661216 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.662120 16636 raft_consensus.cc:2379] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 4: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:14.655915 16633 raft_consensus.cc:2454] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
I20260810 18:52:14.664084 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.664906 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.664930 16633 raft_consensus.cc:2454] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 5.
I20260810 18:52:14.665714 16636 raft_consensus.cc:2454] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
I20260810 18:52:14.610618 20318 raft_consensus.cc:3037] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:14.666846 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.649358 20037 raft_consensus.cc:2726] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260810 18:52:14.670456 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 78294.4us
I20260810 18:52:14.598582 20282 raft_consensus.cc:504] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:14.671447 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:14.659341 19943 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 3.207s user 0.025s sys 0.067s
I20260810 18:52:14.657739 19858 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:14.598083 20110 consensus_queue.cc:1035] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.064s
I20260810 18:52:14.657750 16621 raft_consensus.cc:2454] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
W20260810 18:52:14.687103 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 63332us
I20260810 18:52:14.682286 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.688263 16701 leader_election.cc:302] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:14.666764 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.690346 16633 raft_consensus.cc:2454] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:52:14.691354 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:14.680809 19943 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 3.229s user 0.025s sys 0.067s
I20260810 18:52:14.692059 16633 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:14.694087 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.692605 16749 raft_consensus.cc:2454] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 5.
I20260810 18:52:14.694207 16448 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:14.683594 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 59434.9us
I20260810 18:52:14.694167 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.696149 16636 raft_consensus.cc:2454] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 3.
I20260810 18:52:14.697677 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.698503 16621 raft_consensus.cc:2454] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:14.696941 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:14.697494 17727 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:14.701442 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.702155 16633 raft_consensus.cc:2454] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 5.
I20260810 18:52:14.703117 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:14.703773 16633 raft_consensus.cc:2362] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6. Re-sending same reply.
I20260810 18:52:14.705222 16636 raft_consensus.cc:2354] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 4. Current term is 5.
I20260810 18:52:14.679085 20301 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:14.708922 16450 leader_election.cc:302] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:14.709344 16701 leader_election.cc:334] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.912s (SENT)
I20260810 18:52:14.707432 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.704726 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:14.682793 20067 consensus_queue.cc:1035] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.026s
I20260810 18:52:14.716252 16447 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:14.717411 20296 raft_consensus.cc:2781] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:14.731663 20296 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:14.732100 20296 raft_consensus.cc:3037] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:14.710666 19858 raft_consensus.cc:2781] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:14.732993 19858 raft_consensus.cc:686] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 5 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:14.733956 19858 consensus_queue.cc:237] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:14.719756 16450 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:14.710513 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.735275 16621 raft_consensus.cc:2454] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:14.736161 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.736728 16621 raft_consensus.cc:2454] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:14.737519 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.738092 16621 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 5.
I20260810 18:52:14.730288 16636 raft_consensus.cc:2454] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:14.739027 16633 raft_consensus.cc:2354] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3.
I20260810 18:52:14.720098 19899 raft_consensus.cc:2781] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:52:14.738945 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.855477 16621 raft_consensus.cc:2454] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 3.
I20260810 18:52:14.856523 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.857540 16621 raft_consensus.cc:2396] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20260810 18:52:14.710491 16701 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:14.725833 20332 log.cc:924] Time spent T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.076s user 0.001s sys 0.000s
I20260810 18:52:14.741586 20343 consensus_queue.cc:1046] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:52:14.847636 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.685388 20037 consensus_queue.cc:1035] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260810 18:52:14.851876 16752 raft_consensus.cc:1261] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260810 18:52:14.871460 16633 raft_consensus.cc:2396] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260810 18:52:14.867453 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 151255us
I20260810 18:52:14.860940 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.872165 16621 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 5.
I20260810 18:52:14.872946 16447 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:14.873178 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.873992 16621 raft_consensus.cc:2454] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:14.874930 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.875787 16621 raft_consensus.cc:2379] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 2: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:14.876578 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.879580 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.880234 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:14.880837 16621 raft_consensus.cc:2454] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
I20260810 18:52:14.883450 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:14.883809 16447 leader_election.cc:302] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:14.885572 16621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 1634) took 1100 ms. Trace:
I20260810 18:52:14.886062 16621 rpcz_store.cc:276] 0810 18:52:13.784784 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:14.882139 (+1097355us) service_pool.cc:227] Handling call
0810 18:52:14.885511 (+ 3372us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:14.848593 20308 raft_consensus.cc:2229] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Raft consensus shutting down.
I20260810 18:52:14.850641 20095 raft_consensus.cc:3032] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Stepping down as leader of term 4
I20260810 18:52:14.847663 19684 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:14.858090 19899 raft_consensus.cc:482] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:14.900276 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:14.862123 20023 raft_consensus.cc:2726] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:14.869365 20184 raft_consensus.cc:2781] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:52:14.898619 16504 raft_consensus.cc:1226] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 4. Current term is 5. Ops: []
W20260810 18:52:14.883338 20337 log.cc:924] Time spent T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.211s user 0.000s sys 0.006s
I20260810 18:52:14.920172 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:14.850821 20345 raft_consensus.cc:2726] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:14.924744 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:14.903898 20308 raft_consensus.cc:2258] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Raft consensus is shut down!
I20260810 18:52:14.906935 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:14.910137 20343 raft_consensus.cc:2726] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:14.911466 20282 leader_election.cc:288] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:14.896396 20095 raft_consensus.cc:729] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:14.906620 19684 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:14.918833 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:14.922681 16626 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1457) took 2703 ms (client timeout 2138 ms). Trace:
W20260810 18:52:14.950667 16626 rpcz_store.cc:269] 0810 18:52:12.219377 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:14.922515 (+2703138us) service_pool.cc:214] Skipping call since client already timed out
0810 18:52:14.922619 (+ 104us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260810 18:52:14.922909 17770 raft_consensus.cc:2396] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260810 18:52:14.901358 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:14.925480 16748 raft_consensus.cc:2362] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4. Re-sending same reply.
I20260810 18:52:14.928038 16752 raft_consensus.cc:1261] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260810 18:52:14.957062 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:14.959415 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:14.935009 16746 raft_consensus.cc:1261] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260810 18:52:14.919735 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:14.914462 20296 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:14.906296 19899 raft_consensus.cc:3037] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:14.946771 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:14.928821 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:14.949687 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:14.961704 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:14.925462 20249 raft_consensus.cc:2726] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260810 18:52:14.963092 16748 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43738 (request call id 1649) took 3316 ms (client timeout 1738 ms). Trace:
W20260810 18:52:14.967191 16748 rpcz_store.cc:269] 0810 18:52:11.639853 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:11.640001 (+ 148us) service_pool.cc:227] Handling call
0810 18:52:14.924608 (+3284607us) spinlock_profiling.cc:243] Waited 3.74 ms on lock 0x7b5800270f98. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed03c1b0 00007f1aed03ba94 00007f1ae6723c04 00007f1ae6723b92
0810 18:52:14.955870 (+ 31262us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8604416}
I20260810 18:52:14.919199 20035 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:14.970808 17780 raft_consensus.cc:1261] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260810 18:52:14.971657 20035 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:14.972455 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:14.972976 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:14.973434 16636 raft_consensus.cc:2396] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20260810 18:52:14.974411 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:14.965664 20298 raft_consensus.cc:2726] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:14.975996 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:14.977438 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:14.966624 16626 raft_consensus.cc:1261] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260810 18:52:14.980741 16626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1619) took 1455 ms. Trace:
I20260810 18:52:14.981211 16626 rpcz_store.cc:276] 0810 18:52:13.525180 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:14.951530 (+1426350us) service_pool.cc:227] Handling call
0810 18:52:14.964841 (+ 13311us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:14.980459 (+ 15618us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:14.980668 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 18:52:15.007655 17726 leader_election.cc:334] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.990s (SENT)
I20260810 18:52:15.008335 17726 leader_election.cc:302] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:14.974941 16636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1620) took 1449 ms. Trace:
I20260810 18:52:15.012527 16636 rpcz_store.cc:276] 0810 18:52:13.525556 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:14.974103 (+1448547us) service_pool.cc:227] Handling call
0810 18:52:14.974402 (+ 299us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:14.974601 (+ 199us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:14.974632 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:14.974676 (+ 44us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:14.974717 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:14.974776 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:14.974886 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:15.014814 16636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1622) took 1488 ms. Trace:
I20260810 18:52:15.015350 16636 rpcz_store.cc:276] 0810 18:52:13.526262 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.013824 (+1487562us) service_pool.cc:227] Handling call
0810 18:52:15.014215 (+ 391us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:15.014426 (+ 211us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:15.014460 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:15.014516 (+ 56us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:15.014563 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:15.014630 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:15.014753 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:15.018428 16636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1623) took 1491 ms. Trace:
I20260810 18:52:14.979444 16500 raft_consensus.cc:2454] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 5.
W20260810 18:52:15.020388 20312 log.cc:924] Time spent T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.917s user 0.000s sys 0.008s
I20260810 18:52:14.949163 20184 raft_consensus.cc:686] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
W20260810 18:52:15.021320 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 54700us
I20260810 18:52:15.016842 16626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1621) took 1491 ms. Trace:
I20260810 18:52:15.045562 16626 rpcz_store.cc:276] 0810 18:52:13.525903 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:14.982388 (+1456485us) service_pool.cc:227] Handling call
0810 18:52:14.982790 (+ 402us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:14.983004 (+ 214us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:14.983042 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:15.016487 (+ 33445us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:15.016552 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:15.016633 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:15.016775 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 18:52:15.050539 16447 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.14.32.196:33387 timed out after 2.752s (SENT) [suppressed 44 similar messages]
I20260810 18:52:14.949812 20183 consensus_queue.cc:1046] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.071s
I20260810 18:52:15.029425 20318 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:15.031345 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:14.935894 20278 consensus_queue.cc:1046] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.075s
I20260810 18:52:15.032660 20035 raft_consensus.cc:2726] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:15.033761 16500 raft_consensus.cc:1226] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:15.035089 19945 consensus_queue.cc:1035] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:52:15.036432 20308 ts_tablet_manager.cc:1905] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 18:52:15.051430 17724 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:15.060245 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:15.024717 16636 rpcz_store.cc:276] 0810 18:52:13.526610 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.017498 (+1490888us) service_pool.cc:227] Handling call
0810 18:52:15.017846 (+ 348us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:15.018065 (+ 219us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:15.018099 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:15.018147 (+ 48us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:15.018195 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:15.018256 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:15.018370 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 18:52:15.114333 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 62982.2us
I20260810 18:52:15.115098 17724 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:15.115305 16636 raft_consensus.cc:1261] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260810 18:52:15.115912 16636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1625) took 1533 ms. Trace:
I20260810 18:52:15.116225 16636 rpcz_store.cc:276] 0810 18:52:13.582579 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.114645 (+1532066us) service_pool.cc:227] Handling call
0810 18:52:15.114961 (+ 316us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:15.115650 (+ 689us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:15.115844 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:15.066385 16626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1624) took 1515 ms. Trace:
I20260810 18:52:15.119189 16626 rpcz_store.cc:276] 0810 18:52:13.551475 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.046775 (+1495300us) service_pool.cc:227] Handling call
0810 18:52:15.065761 (+ 18986us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:15.065989 (+ 228us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:15.066024 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:15.066078 (+ 54us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:15.066126 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:15.066193 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:15.066316 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:15.029484 17775 raft_consensus.cc:1261] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260810 18:52:15.033762 19684 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:15.020936 20296 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:15.137105 19945 consensus_queue.cc:1225] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer 980f343dd1fd4e049143f8b9d9f2f1be log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260810 18:52:15.150102 16500 raft_consensus.cc:1226] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:15.180416 16495 raft_consensus.cc:2379] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 5: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
W20260810 18:52:15.157032 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 80619.3us
I20260810 18:52:15.150605 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:15.184132 16504 raft_consensus.cc:3037] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:15.191634 17775 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55570 (request call id 433) took 1130 ms. Trace:
I20260810 18:52:15.192212 17775 rpcz_store.cc:276] 0810 18:52:14.061553 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.028700 (+967147us) service_pool.cc:227] Handling call
0810 18:52:15.029055 (+ 355us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:15.138700 (+109645us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:15.190329 (+ 51629us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:15.193845 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:15.194082 20278 consensus_queue.cc:1225] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer 831925dd8b644ebab7eb0d672a94eddd log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260810 18:52:15.193998 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:15.194752 16628 raft_consensus.cc:2354] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for earlier term 4. Current term is 5.
I20260810 18:52:15.196594 16628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 170) took 1166 ms. Trace:
I20260810 18:52:15.196918 16628 rpcz_store.cc:276] 0810 18:52:14.030442 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.193511 (+1163069us) service_pool.cc:227] Handling call
0810 18:52:15.195197 (+ 1686us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:15.197516 17778 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55586 (request call id 331) took 4429 ms. Trace:
I20260810 18:52:15.198263 16628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1632) took 1418 ms. Trace:
I20260810 18:52:15.197898 17778 rpcz_store.cc:276] 0810 18:52:10.768203 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:10.881231 (+113028us) service_pool.cc:227] Handling call
0810 18:52:10.887628 (+ 6397us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:11.279630 (+392002us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:11.280613 (+ 983us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:11.283464 (+ 2851us) raft_consensus.cc:1631] Marking committed up to 6
0810 18:52:11.289187 (+ 5723us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:15.197268 (+3908081us) spinlock_profiling.cc:243] Waited 382 ms on lock 0x7b54000348d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:15.197312 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:15.197459 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":877797760}
I20260810 18:52:15.198606 16628 rpcz_store.cc:276] 0810 18:52:13.780095 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.197515 (+1417420us) service_pool.cc:227] Handling call
0810 18:52:15.197792 (+ 277us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:15.197967 (+ 175us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:15.197996 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:15.198035 (+ 39us) raft_consensus.cc:1631] Marking committed up to 6
0810 18:52:15.198063 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:15.198113 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:15.198215 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:15.201399 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:15.201821 16628 raft_consensus.cc:1261] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260810 18:52:15.202371 16628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1633) took 1422 ms. Trace:
I20260810 18:52:15.202656 16628 rpcz_store.cc:276] 0810 18:52:13.780412 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.201082 (+1420670us) service_pool.cc:227] Handling call
0810 18:52:15.201376 (+ 294us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:15.202174 (+ 798us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:15.202328 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:15.204047 16628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1635) took 1415 ms. Trace:
I20260810 18:52:15.204355 16628 rpcz_store.cc:276] 0810 18:52:13.788745 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.203347 (+1414602us) service_pool.cc:227] Handling call
0810 18:52:15.203613 (+ 266us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:15.203773 (+ 160us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:15.203794 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:15.203841 (+ 47us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:15.203868 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:15.203915 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:15.203999 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:15.205897 16628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1636) took 1407 ms. Trace:
I20260810 18:52:15.206192 16628 rpcz_store.cc:276] 0810 18:52:13.798947 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.205179 (+1406232us) service_pool.cc:227] Handling call
0810 18:52:15.205443 (+ 264us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:15.205605 (+ 162us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:15.205629 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:15.205667 (+ 38us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:15.205695 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:15.205744 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:15.205854 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:15.207528 16628 raft_consensus.cc:1261] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260810 18:52:15.214740 16752 raft_consensus.cc:1261] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260810 18:52:15.217645 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:15.179174 20297 raft_consensus.cc:2781] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:15.227625 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:15.229218 16622 raft_consensus.cc:2454] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:15.233915 16740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43726 (request call id 1418) took 1313 ms. Trace:
I20260810 18:52:15.234390 16740 rpcz_store.cc:276] 0810 18:52:13.920925 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:13.945115 (+ 24190us) service_pool.cc:227] Handling call
0810 18:52:13.945499 (+ 384us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:13.953731 (+ 8232us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:13.953771 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:13.954299 (+ 528us) log.cc:842] Serialized 26 byte log entry
0810 18:52:14.086272 (+131973us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:52:14.086308 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:14.086371 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:14.750132 (+663761us) spinlock_profiling.cc:243] Waited 5.1 ms on lock 0x7b44000232cc. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae50e4a95 00007f1ae50e54b2 00007f1ae7d8d511 00007f1ae7d88e1a 00007f1ae7d7cbee 00007f1ae7d7bf36 00007f1ae7d7bb58 00007f1ae7d85551 00007f1ae7d85509 00007f1ae7d85499
0810 18:52:15.233589 (+483457us) raft_consensus.cc:1671] finished
0810 18:52:15.233610 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:15.233825 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11732224,"thread_start_us":102722,"threads_started":1,"wal-append.queue_time_us":128633}
W20260810 18:52:15.249568 16572 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53870 dropped due to backpressure. The service queue is full; it has 50 items. [suppressed 7 similar messages]
I20260810 18:52:15.250152 16572 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53870 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
I20260810 18:52:15.256669 16504 raft_consensus.cc:2454] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 5.
I20260810 18:52:15.232304 16743 pending_rounds.cc:76] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be: Aborting all ops after (but not including) 1
I20260810 18:52:15.293128 16743 pending_rounds.cc:97] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260810 18:52:15.293440 16743 raft_consensus.cc:2866] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:52:15.293764 16743 pending_rounds.cc:97] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260810 18:52:15.227357 19561 consensus_queue.cc:1035] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.067s
I20260810 18:52:15.293989 16743 raft_consensus.cc:2866] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:52:15.261492 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:15.211026 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:15.296037 16622 raft_consensus.cc:2454] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 2.
I20260810 18:52:15.296898 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:15.297437 16622 raft_consensus.cc:3037] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:15.262434 16628 raft_consensus.cc:1226] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:15.268083 16740 raft_consensus.cc:1261] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260810 18:52:15.291394 20184 consensus_queue.cc:237] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:15.299345 17726 leader_election.cc:334] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.894s (SENT)
I20260810 18:52:15.298741 16448 leader_election.cc:302] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:15.301518 16746 raft_consensus.cc:1261] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260810 18:52:15.232587 20297 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:15.307971 20035 consensus_queue.cc:1035] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20260810 18:52:15.308758 20035 consensus_queue.cc:1225] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer 980f343dd1fd4e049143f8b9d9f2f1be log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260810 18:52:15.315642 16622 raft_consensus.cc:2454] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:15.304431 16628 raft_consensus.cc:3032] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 LEADER]: Stepping down as leader of term 6
I20260810 18:52:15.316360 16628 raft_consensus.cc:729] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:15.228432 20096 consensus_queue.cc:1035] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.071s
I20260810 18:52:15.226893 19899 raft_consensus.cc:504] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:15.326709 16746 pending_rounds.cc:76] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be: Aborting all ops after (but not including) 4
I20260810 18:52:15.347785 16746 pending_rounds.cc:97] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260810 18:52:15.348412 16746 raft_consensus.cc:2866] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:52:15.341897 16622 raft_consensus.cc:3037] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
W20260810 18:52:15.342350 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 43684.5us
I20260810 18:52:15.306977 20297 raft_consensus.cc:3037] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:15.317364 16628 consensus_queue.cc:260] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:15.351022 17787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55602 (request call id 349) took 1308 ms. Trace:
I20260810 18:52:15.362749 17787 rpcz_store.cc:276] 0810 18:52:14.043082 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:14.043229 (+ 147us) service_pool.cc:227] Handling call
0810 18:52:14.043481 (+ 252us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:14.525427 (+481946us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:15.350707 (+825280us) spinlock_profiling.cc:243] Waited 298 ms on lock 0x7b540072e554. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:15.350946 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":684417792}
I20260810 18:52:15.364516 16628 raft_consensus.cc:3037] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:15.352866 16448 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:15.378257 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:15.390483 16628 raft_consensus.cc:1261] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20260810 18:52:15.400072 16749 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.676s user 0.012s sys 0.063s
W20260810 18:52:15.400595 16749 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.676s user 0.012s sys 0.064s
W20260810 18:52:15.401674 17783 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.677s user 0.014s sys 0.057s
W20260810 18:52:15.404161 17783 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.680s user 0.014s sys 0.057s
I20260810 18:52:15.417086 16502 raft_consensus.cc:2354] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3.
I20260810 18:52:15.345824 20319 raft_consensus.cc:2726] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260810 18:52:15.372978 19943 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.644s user 0.033s sys 0.042s
I20260810 18:52:15.346989 20318 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:15.431239 16628 raft_consensus.cc:3032] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:15.438920 16628 raft_consensus.cc:729] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:15.439915 16628 consensus_queue.cc:260] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:15.441097 16628 raft_consensus.cc:3037] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
W20260810 18:52:15.418380 20316 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.319s user 0.008s sys 0.032s
I20260810 18:52:15.372342 19561 raft_consensus.cc:2726] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260810 18:52:15.461241 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 45454us
I20260810 18:52:15.413921 19863 raft_consensus.cc:2726] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260810 18:52:15.428318 20256 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.705s user 0.010s sys 0.028s
W20260810 18:52:15.437714 19943 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.720s user 0.033s sys 0.042s
I20260810 18:52:15.508519 16628 raft_consensus.cc:1261] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260810 18:52:15.511711 16622 raft_consensus.cc:1261] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260810 18:52:15.465267 19480 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
W20260810 18:52:15.470937 20256 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.751s user 0.010s sys 0.029s
W20260810 18:52:15.470718 19963 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.004s sys 0.016s
I20260810 18:52:15.414530 20190 consensus_queue.cc:1035] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
I20260810 18:52:15.466755 19658 consensus_queue.cc:1035] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.048s
W20260810 18:52:15.512511 20345 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.028s sys 0.037s
W20260810 18:52:15.520327 16701 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.662s user 0.006s sys 0.026s
W20260810 18:52:15.551936 16701 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.693s user 0.007s sys 0.026s
W20260810 18:52:15.525291 16633 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.622s user 0.002s sys 0.015s
W20260810 18:52:15.552838 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 74047.8us
W20260810 18:52:15.553334 16633 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.651s user 0.002s sys 0.016s
W20260810 18:52:15.528724 16573 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.677s user 0.016s sys 0.037s
W20260810 18:52:15.528720 16745 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.566s user 0.003s sys 0.018s
W20260810 18:52:15.554039 16573 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.703s user 0.017s sys 0.037s
W20260810 18:52:15.554381 16745 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.592s user 0.003s sys 0.018s
W20260810 18:52:15.552935 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 843549us
W20260810 18:52:15.566967 16701 leader_election.cc:334] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.032s (SENT)
W20260810 18:52:15.573524 16695 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.694s user 0.010s sys 0.021s
W20260810 18:52:15.573921 16695 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.695s user 0.010s sys 0.021s
I20260810 18:52:15.574162 16573 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:15.467347 20316 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.368s user 0.009s sys 0.033s
W20260810 18:52:15.574179 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 702157us
W20260810 18:52:15.580837 16750 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.720s user 0.000s sys 0.023s
W20260810 18:52:15.581624 16750 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.726s user 0.001s sys 0.023s
W20260810 18:52:15.530133 19858 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.793s user 0.003s sys 0.023s
W20260810 18:52:15.551818 20345 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.627s user 0.028s sys 0.037s
W20260810 18:52:15.527529 19963 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.637s user 0.004s sys 0.016s
W20260810 18:52:15.599838 16621 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.704s user 0.006s sys 0.022s
I20260810 18:52:15.601861 18728 consensus_queue.cc:1046] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260810 18:52:15.602232 16448 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:15.603663 16742 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.663s user 0.012s sys 0.017s
W20260810 18:52:15.604092 16742 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.663s user 0.012s sys 0.017s
W20260810 18:52:15.585255 19858 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.848s user 0.004s sys 0.023s
I20260810 18:52:15.593003 17770 raft_consensus.cc:1226] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:15.594225 16748 raft_consensus.cc:3032] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 LEADER]: Stepping down as leader of term 4
I20260810 18:52:15.607784 16748 raft_consensus.cc:729] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:15.524413 19480 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:15.591599 16701 leader_election.cc:334] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.133s (SENT)
I20260810 18:52:15.582083 19391 raft_consensus.cc:2726] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:15.608811 16748 consensus_queue.cc:260] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:15.621471 16748 raft_consensus.cc:3037] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:15.581897 20316 raft_consensus.cc:686] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
W20260810 18:52:15.605232 16621 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.715s user 0.007s sys 0.022s
W20260810 18:52:15.616586 16701 leader_election.cc:334] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.171s (SENT)
I20260810 18:52:15.623595 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:15.552261 19943 consensus_queue.cc:1035] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.051s
I20260810 18:52:15.618101 16628 raft_consensus.cc:1261] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260810 18:52:15.618829 17726 leader_election.cc:334] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.445s (SENT)
I20260810 18:52:15.624682 17726 leader_election.cc:302] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:15.624162 16622 raft_consensus.cc:3037] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
W20260810 18:52:15.606364 20343 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.665s user 0.017s sys 0.020s
I20260810 18:52:15.636582 18728 consensus_queue.cc:1035] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
W20260810 18:52:15.611299 19786 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.632s user 0.017s sys 0.024s
I20260810 18:52:15.608098 20164 raft_consensus.cc:2781] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:15.642933 16622 raft_consensus.cc:2454] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:15.656025 16448 leader_election.cc:302] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:15.593549 19351 consensus_queue.cc:1046] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260810 18:52:15.580643 20249 consensus_queue.cc:1035] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
W20260810 18:52:15.662920 16447 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.609s user 0.000s sys 0.049s
W20260810 18:52:15.670187 16447 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.617s user 0.000s sys 0.050s
W20260810 18:52:15.670620 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 620073us
W20260810 18:52:15.630064 20343 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.689s user 0.018s sys 0.020s
W20260810 18:52:15.656771 19786 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.665s user 0.017s sys 0.024s
I20260810 18:52:15.632973 20301 raft_consensus.cc:2726] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260810 18:52:15.705003 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 49227.2us
W20260810 18:52:15.705785 17726 leader_election.cc:334] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.195s (SENT)
I20260810 18:52:15.707603 17726 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:15.706494 16626 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.017s sys 0.034s
W20260810 18:52:15.709372 16626 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.531s user 0.017s sys 0.034s
I20260810 18:52:15.627528 19907 consensus_queue.cc:1035] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.081s
I20260810 18:52:15.662390 19899 leader_election.cc:288] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 3 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:15.620929 19935 consensus_queue.cc:1035] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.069s
I20260810 18:52:15.643971 19905 consensus_queue.cc:1035] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.065s
I20260810 18:52:15.670154 20164 raft_consensus.cc:686] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:15.729092 16748 raft_consensus.cc:1261] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260810 18:52:15.729952 16748 pending_rounds.cc:76] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be: Aborting all ops after (but not including) 2
I20260810 18:52:15.730298 16748 pending_rounds.cc:97] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260810 18:52:15.730706 16748 raft_consensus.cc:2866] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:52:15.667368 19595 consensus_queue.cc:1035] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
W20260810 18:52:15.730986 16574 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.757s user 0.017s sys 0.019s
I20260810 18:52:15.722167 19912 consensus_queue.cc:1035] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260810 18:52:15.723629 20035 raft_consensus.cc:2726] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:15.721107 19996 raft_consensus.cc:2781] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election won for term 3
W20260810 18:52:15.737393 16574 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.869s user 0.018s sys 0.019s
W20260810 18:52:15.761490 20344 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.994s user 0.042s sys 0.020s
I20260810 18:52:15.774511 16508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1422) took 3136 ms. Trace:
I20260810 18:52:15.769730 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:15.776921 16748 raft_consensus.cc:3037] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:15.776634 16508 rpcz_store.cc:276] 0810 18:52:12.638231 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:12.639096 (+ 865us) service_pool.cc:227] Handling call
0810 18:52:12.708300 (+ 69204us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:12.855535 (+147235us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:15.774248 (+2918713us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b54001ff9d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:15.774451 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":336483584}
I20260810 18:52:15.785877 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:15.791321 16500 raft_consensus.cc:1226] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 4. Current term is 5. Ops: []
W20260810 18:52:15.773729 20344 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.007s user 0.042s sys 0.020s
I20260810 18:52:15.795825 20164 consensus_queue.cc:237] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:15.734509 20316 consensus_queue.cc:237] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:15.797711 20379 log.cc:924] Time spent T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
W20260810 18:52:15.799686 20261 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260810 18:52:15.763605 19996 raft_consensus.cc:686] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:15.802284 19996 consensus_queue.cc:237] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:15.803812 19935 consensus_queue.cc:1035] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260810 18:52:15.806111 20378 log.cc:924] Time spent T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20260810 18:52:15.807998 16447 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:15.809124 20362 consensus_queue.cc:1035] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 18:52:15.810127 20239 raft_consensus.cc:2726] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:15.765453 20367 raft_consensus.cc:482] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:15.812942 20210 log.cc:924] Time spent T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 4.089s user 0.004s sys 0.020s
I20260810 18:52:15.760908 20364 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:15.812481 20367 raft_consensus.cc:504] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:15.790591 19480 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:15.763520 17780 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s user 0.031s sys 0.039s
I20260810 18:52:15.814528 16623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1601) took 4565 ms. Trace:
W20260810 18:52:15.826617 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 202779us
I20260810 18:52:15.824285 16623 rpcz_store.cc:276] 0810 18:52:11.248993 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:11.249147 (+ 154us) service_pool.cc:227] Handling call
0810 18:52:11.254927 (+ 5780us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:11.255283 (+ 356us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:11.255316 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:11.255924 (+ 608us) log.cc:842] Serialized 26 byte log entry
0810 18:52:11.489355 (+233431us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:11.489399 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:11.489468 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:15.812305 (+4322837us) spinlock_profiling.cc:243] Waited 8.04 ms on lock 0x7b44000232cc. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae50e4a95 00007f1ae50e54b2 00007f1ae7d8d511 00007f1ae7d88e1a 00007f1ae7d7cbee 00007f1ae7d7bf36 00007f1ae7d7bb58 00007f1ae7d85551 00007f1ae7d85509 00007f1ae7d85499
0810 18:52:15.814220 (+ 1915us) raft_consensus.cc:1671] finished
0810 18:52:15.814253 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:15.814463 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":221,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":90,"spinlock_wait_cycles":18496384,"thread_start_us":232499,"threads_started":1,"wal-append.queue_time_us":411634}
I20260810 18:52:15.833583 20245 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:15.834233 20245 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:15.834991 17726 leader_election.cc:334] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.723s (SENT)
I20260810 18:52:15.835480 17726 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:15.835806 20345 raft_consensus.cc:482] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:15.836524 20035 raft_consensus.cc:2726] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:15.836350 20345 raft_consensus.cc:504] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:15.837886 16500 raft_consensus.cc:1226] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260810 18:52:15.841163 20348 consensus_queue.cc:1046] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:52:15.842792 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:15.839680 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:15.851218 17724 leader_election.cc:302] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:15.852476 16748 raft_consensus.cc:2454] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:15.853557 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:15.838763 20245 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:15.736054 20369 consensus_queue.cc:1046] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.082s
I20260810 18:52:15.866160 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:15.870077 16623 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
I20260810 18:52:15.813512 20364 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:15.828852 17780 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.650s user 0.032s sys 0.039s
I20260810 18:52:15.826319 16636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1628) took 2073 ms. Trace:
W20260810 18:52:15.868983 16701 leader_election.cc:334] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.403s (SENT)
I20260810 18:52:15.871423 16448 leader_election.cc:302] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:15.875654 16701 leader_election.cc:302] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:15.881660 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:15.883391 16448 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:15.838871 20345 leader_election.cc:288] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:15.866808 19858 raft_consensus.cc:2781] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:15.874974 16636 rpcz_store.cc:276] 0810 18:52:13.752746 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.117086 (+1364340us) service_pool.cc:227] Handling call
0810 18:52:15.117446 (+ 360us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:15.117688 (+ 242us) raft_consensus.cc:1520] Early marking committed up to index 2
0810 18:52:15.615944 (+498256us) log.cc:842] Serialized 14 byte log entry
0810 18:52:15.825977 (+210033us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:15.826029 (+ 52us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:15.826077 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:15.826141 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:15.826257 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":107,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":24029,"thread_start_us":635669,"threads_started":2}
I20260810 18:52:15.924549 20150 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:15.950207 17726 leader_election.cc:334] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.868s (SENT)
I20260810 18:52:15.857261 20298 consensus_queue.cc:1046] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.089s
I20260810 18:52:16.001260 17775 raft_consensus.cc:1226] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:16.219972 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:17.125823 17775 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55570 (request call id 448) took 1271 ms. Trace:
I20260810 18:52:17.127672 17726 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:16.323431 20076 consensus_queue.cc:1046] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 1.272s
I20260810 18:52:17.126840 17780 raft_consensus.cc:2396] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20260810 18:52:16.859026 16622 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.126s user 0.016s sys 0.011s
W20260810 18:52:17.136524 16622 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.404s user 0.022s sys 0.015s
W20260810 18:52:16.436164 20344 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.640s user 0.006s sys 0.021s
W20260810 18:52:16.862808 16621 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.031s user 0.003s sys 0.004s
W20260810 18:52:17.137897 20344 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.342s user 0.006s sys 0.021s
W20260810 18:52:17.138119 16621 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.306s user 0.005s sys 0.006s
W20260810 18:52:16.869009 16628 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.122s user 0.004s sys 0.026s
W20260810 18:52:17.139045 16628 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.392s user 0.010s sys 0.036s
I20260810 18:52:16.821031 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260810 18:52:17.139887 20384 log.cc:924] Time spent T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.921s user 0.002s sys 0.003s
W20260810 18:52:16.502828 20387 log.cc:924] Time spent T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.518s user 0.000s sys 0.000s
I20260810 18:52:16.001482 20150 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:16.704365 20297 consensus_meta.cc:287] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 1.282s user 0.011s sys 0.002s
I20260810 18:52:17.142797 20297 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:16.176640 20395 log.cc:924] Time spent T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.453s user 0.000s sys 0.001s
W20260810 18:52:16.845799 16505 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.055s user 0.001s sys 0.035s
I20260810 18:52:15.944579 19943 raft_consensus.cc:2726] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260810 18:52:16.927285 16742 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.130s user 0.027s sys 0.022s
W20260810 18:52:17.156900 16505 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.366s user 0.005s sys 0.043s
I20260810 18:52:15.947749 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
W20260810 18:52:17.175464 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 1.22521e+06us
I20260810 18:52:17.175907 17781 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55570 (request call id 449) took 1321 ms. Trace:
I20260810 18:52:16.557626 20308 ts_tablet_manager.cc:1918] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260810 18:52:17.176602 17781 rpcz_store.cc:276] 0810 18:52:15.854802 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.857410 (+ 2608us) service_pool.cc:227] Handling call
0810 18:52:17.175656 (+1318246us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:17.177759 20308 log.cc:1196] T cdfc652dd163415b94d3f0bc3931a07f P 0397d760589d40d3b92ef141cdfc2dbb: Deleting WAL directory at /tmp/dist-test-taskG_N3N6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786387803181742-14467-0/minicluster-data/ts-1-root/wals/cdfc652dd163415b94d3f0bc3931a07f
I20260810 18:52:17.178009 20150 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:17.126510 17775 rpcz_store.cc:276] 0810 18:52:15.854439 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.857199 (+ 2760us) service_pool.cc:227] Handling call
0810 18:52:15.945652 (+ 88453us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:17.124817 (+1179165us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:17.125462 (+ 645us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:17.139936 20035 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:17.183224 16742 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.386s user 0.027s sys 0.026s
I20260810 18:52:16.493777 20367 leader_election.cc:288] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:17.192245 20035 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:17.199611 20297 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:17.201002 20035 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:17.200564 20095 consensus_queue.cc:260] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:17.201454 16752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1706) took 1981 ms. Trace:
I20260810 18:52:17.202350 20095 raft_consensus.cc:3037] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:17.203426 16633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1614) took 3692 ms. Trace:
W20260810 18:52:16.503613 20401 log.cc:924] Time spent T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.353s user 0.000s sys 0.001s
W20260810 18:52:16.258284 20184 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.044s sys 0.112s
W20260810 18:52:17.155854 19786 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.629s user 0.007s sys 0.015s
I20260810 18:52:17.205097 16495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52352 (request call id 1471) took 2286 ms. Trace:
W20260810 18:52:17.226889 16448 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.242s user 0.029s sys 0.034s
W20260810 18:52:17.227352 19786 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.283s user 0.008s sys 0.015s
W20260810 18:52:17.227762 16448 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.243s user 0.030s sys 0.035s
I20260810 18:52:17.228179 19786 raft_consensus.cc:3032] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 4 LEADER]: Stepping down as leader of term 4
W20260810 18:52:17.228623 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 1.28043e+06us
I20260810 18:52:17.228889 19786 raft_consensus.cc:729] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:17.229553 20367 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:16.176621 20371 consensus_queue.cc:1046] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 1.286s
I20260810 18:52:17.230896 19786 consensus_queue.cc:260] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:17.231882 16506 raft_consensus.cc:1226] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:17.233359 20369 raft_consensus.cc:3032] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:17.233938 20369 raft_consensus.cc:729] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:17.202020 16752 rpcz_store.cc:276] 0810 18:52:15.219816 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.219984 (+ 168us) service_pool.cc:227] Handling call
0810 18:52:15.220347 (+ 363us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:15.220772 (+ 425us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:15.220810 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:15.240912 (+ 20102us) log.cc:842] Serialized 26 byte log entry
0810 18:52:15.736560 (+495648us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:52:15.736610 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:15.736673 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:17.185944 (+1449271us) raft_consensus.cc:1671] finished
0810 18:52:17.200871 (+ 14927us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:17.201151 (+ 280us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4329,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":10357,"thread_start_us":397142,"threads_started":1,"wal-append.queue_time_us":811849}
W20260810 18:52:17.237435 20396 log.cc:924] Time spent T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 1.426s user 0.018s sys 0.005s
I20260810 18:52:17.238917 16752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 244) took 1428 ms. Trace:
I20260810 18:52:17.239377 16752 rpcz_store.cc:276] 0810 18:52:15.810025 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:17.237270 (+1427245us) service_pool.cc:227] Handling call
0810 18:52:17.237936 (+ 666us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:17.238431 (+ 495us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:17.238469 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:17.238554 (+ 85us) raft_consensus.cc:1631] Marking committed up to 6
0810 18:52:17.238608 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:17.238672 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:17.238816 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:17.244184 20369 consensus_queue.cc:260] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:17.246022 20150 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:17.246197 20308 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53868 (request call id 960) took 3367 ms. Trace:
W20260810 18:52:16.504596 20382 log.cc:924] Time spent T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.349s user 0.000s sys 0.000s
W20260810 18:52:16.519491 20410 log.cc:924] Time spent T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.071s user 0.000s sys 0.001s
W20260810 18:52:16.130317 20377 log.cc:924] Time spent T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.387s user 0.000s sys 0.001s
W20260810 18:52:16.765193 19858 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.659s user 0.012s sys 0.006s
I20260810 18:52:17.203979 16633 rpcz_store.cc:276] 0810 18:52:13.510743 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:14.888581 (+1377838us) service_pool.cc:227] Handling call
0810 18:52:14.888915 (+ 334us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:15.554042 (+665127us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:15.556342 (+ 2300us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:15.584661 (+ 28319us) log.cc:842] Serialized 26 byte log entry
0810 18:52:15.882145 (+297484us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:15.882184 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:15.882247 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:17.186859 (+1304612us) raft_consensus.cc:1671] finished
0810 18:52:17.202705 (+ 15846us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:17.203030 (+ 325us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":30557,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":173,"thread_start_us":903728,"threads_started":2,"wal-append.queue_time_us":669960}
W20260810 18:52:17.254876 20184 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.514s user 0.044s sys 0.129s
W20260810 18:52:17.287451 19858 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.220s user 0.012s sys 0.008s
W20260810 18:52:17.269762 17727 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.156s user 0.003s sys 0.003s
I20260810 18:52:17.287957 19858 raft_consensus.cc:686] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 5 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:17.273952 20376 raft_consensus.cc:2726] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:17.274247 20150 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:16.033396 16623 raft_consensus.cc:3032] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:52:17.298410 16623 raft_consensus.cc:729] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:17.275359 19480 raft_consensus.cc:482] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:17.300164 20369 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:17.300799 19480 raft_consensus.cc:504] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:17.278467 17780 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55570 (request call id 451) took 1421 ms. Trace:
I20260810 18:52:17.280157 16624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1613) took 3770 ms. Trace:
I20260810 18:52:17.230587 20367 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:17.281790 16752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1727) took 1467 ms. Trace:
I20260810 18:52:17.283294 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:17.307803 16752 rpcz_store.cc:276] 0810 18:52:15.814262 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:17.254669 (+1440407us) service_pool.cc:227] Handling call
0810 18:52:17.280902 (+ 26233us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:17.281395 (+ 493us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:17.281423 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:17.281481 (+ 58us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:17.281518 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:17.281600 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:17.281713 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:17.257537 20308 rpcz_store.cc:276] 0810 18:52:13.891831 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:13.917233 (+ 25402us) service_pool.cc:227] Handling call
0810 18:52:13.998643 (+ 81410us) ts_tablet_manager.cc:1145] Deleting tablet cdfc652dd163415b94d3f0bc3931a07f
0810 18:52:14.001567 (+ 2924us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 18:52:15.800235 (+1798668us) tablet_metadata.cc:641] Metadata flushed
0810 18:52:16.557280 (+757045us) tablet_metadata.cc:641] Metadata flushed
0810 18:52:17.246022 (+688742us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":605858,"tablet-delete.queue_time_us":57835,"thread_start_us":39935,"threads_started":1}
W20260810 18:52:17.288205 17727 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.313s user 0.004s sys 0.003s
I20260810 18:52:17.271267 20298 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:17.192870 16447 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.240s user 0.012s sys 0.022s
I20260810 18:52:17.295418 19976 raft_consensus.cc:2726] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:17.200598 20297 raft_consensus.cc:2726] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:17.284108 20364 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:17.227367 16495 rpcz_store.cc:276] 0810 18:52:14.918769 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:14.919848 (+ 1079us) service_pool.cc:227] Handling call
0810 18:52:17.204579 (+2284731us) spinlock_profiling.cc:243] Waited 259 ms on lock 0x7b54000e1ad4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:17.204815 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":596198144}
I20260810 18:52:17.300050 20150 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:17.300019 16623 consensus_queue.cc:260] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:17.314138 16447 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.362s user 0.013s sys 0.022s
I20260810 18:52:17.297079 19858 consensus_queue.cc:237] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:17.325403 17785 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55586 (request call id 341) took 2476 ms (client timeout 1510 ms). Trace:
I20260810 18:52:17.325758 16626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1629) took 3560 ms. Trace:
W20260810 18:52:17.325963 17785 rpcz_store.cc:269] 0810 18:52:14.849105 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:14.890665 (+ 41560us) service_pool.cc:227] Handling call
0810 18:52:17.282827 (+2392162us) spinlock_profiling.cc:243] Waited 6.98 ms on lock 0x7b5800005498. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed03c1b0 00007f1aed03ba94 00007f1ae6723c04 00007f1ae6723b92
0810 18:52:17.325255 (+ 42428us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16048896}
I20260810 18:52:17.326218 20298 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:17.331873 19929 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:17.349707 19929 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:17.339246 16745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1720) took 1738 ms. Trace:
I20260810 18:52:17.304478 17780 rpcz_store.cc:276] 0810 18:52:15.856618 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:16.154560 (+297942us) service_pool.cc:227] Handling call
0810 18:52:17.259790 (+1105230us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:17.340308 16501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1428) took 4457 ms. Trace:
I20260810 18:52:17.342275 19595 raft_consensus.cc:3032] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 LEADER]: Stepping down as leader of term 4
I20260810 18:52:17.343096 20367 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:17.327811 16626 rpcz_store.cc:276] 0810 18:52:13.765582 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.120399 (+1354817us) service_pool.cc:227] Handling call
0810 18:52:15.137694 (+ 17295us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:15.177488 (+ 39794us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:15.177518 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:15.178018 (+ 500us) log.cc:842] Serialized 26 byte log entry
0810 18:52:15.709843 (+531825us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:15.709889 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:15.709970 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:17.236221 (+1526251us) spinlock_profiling.cc:243] Waited 8.21 ms on lock 0x7b44000232cc. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae50e4a95 00007f1ae50e54b2 00007f1ae7d8d511 00007f1ae7d88e1a 00007f1ae7d7cbee 00007f1ae7d7bf36 00007f1ae7d7bb58 00007f1ae7d85551 00007f1ae7d85509 00007f1ae7d85499
0810 18:52:17.311037 (+ 74816us) raft_consensus.cc:1671] finished
0810 18:52:17.311078 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:17.325574 (+ 14496us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":40692,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":12119,"spinlock_wait_cycles":18879872,"thread_start_us":531421,"threads_started":1,"wal-append.queue_time_us":632843}
W20260810 18:52:17.345665 16636 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.242s user 0.029s sys 0.068s
I20260810 18:52:17.347973 16749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 230) took 2634 ms. Trace:
I20260810 18:52:17.321003 20376 raft_consensus.cc:482] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:17.304728 16624 rpcz_store.cc:276] 0810 18:52:13.510026 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:13.511255 (+ 1229us) service_pool.cc:227] Handling call
0810 18:52:13.511694 (+ 439us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:13.511985 (+ 291us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:13.512021 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:15.206797 (+1694776us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b2c000002e8. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae6adde11 00007f1ae6aed442 00007f1ae8135c11 00007f1ae80fa58a 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772
0810 18:52:15.207180 (+ 383us) log.cc:842] Serialized 26 byte log entry
0810 18:52:15.515335 (+308155us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:15.515391 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:15.515466 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:17.163989 (+1648523us) raft_consensus.cc:1671] finished
0810 18:52:17.178933 (+ 14944us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:17.279951 (+101018us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24504064,"thread_start_us":307043,"threads_started":1,"wal-append.queue_time_us":425949}
I20260810 18:52:17.348351 16747 raft_consensus.cc:1226] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260810 18:52:17.368263 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:17.368641 16748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1726) took 1560 ms. Trace:
I20260810 18:52:17.361063 16749 rpcz_store.cc:276] 0810 18:52:14.713525 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:14.723245 (+ 9720us) service_pool.cc:227] Handling call
0810 18:52:14.723707 (+ 462us) raft_consensus.cc:1457] Updating replica for 2 ops
0810 18:52:15.401099 (+677392us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:15.401142 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0810 18:52:15.416424 (+ 15282us) log.cc:842] Serialized 52 byte log entry
0810 18:52:15.737293 (+320869us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:15.737323 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:15.737391 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:17.347325 (+1609934us) raft_consensus.cc:1671] finished
0810 18:52:17.347369 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:17.347687 (+ 318us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":164723,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":4189,"thread_start_us":945056,"threads_started":2,"wal-append.queue_time_us":427759}
I20260810 18:52:17.358250 16501 rpcz_store.cc:276] 0810 18:52:12.882883 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:13.011895 (+129012us) service_pool.cc:227] Handling call
0810 18:52:13.012194 (+ 299us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:13.492446 (+480252us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:17.339725 (+3847279us) spinlock_profiling.cc:243] Waited 479 ms on lock 0x7b5400117954. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:17.340074 (+ 349us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1102673152}
I20260810 18:52:17.360116 20297 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:17.350986 20298 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:17.374799 20376 raft_consensus.cc:504] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:17.375504 20345 raft_consensus.cc:482] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
W20260810 18:52:17.376260 17724 leader_election.cc:334] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Timed out: RequestConsensusVote RPC to 127.14.32.193:40791 timed out after 2.189s (SENT)
I20260810 18:52:17.358251 16745 rpcz_store.cc:276] 0810 18:52:15.592710 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.596787 (+ 4077us) service_pool.cc:227] Handling call
0810 18:52:15.597231 (+ 444us) raft_consensus.cc:1457] Updating replica for 3 ops
0810 18:52:15.617755 (+ 20524us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:15.617826 (+ 71us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0810 18:52:15.627953 (+ 10127us) log.cc:842] Serialized 78 byte log entry
0810 18:52:15.926801 (+298848us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:16.030649 (+103848us) log.cc:842] Serialized 14 byte log entry
0810 18:52:17.275191 (+1244542us) log.cc:842] Serialized 14 byte log entry
0810 18:52:17.309528 (+ 34337us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:17.319621 (+ 10093us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:17.323167 (+ 3546us) raft_consensus.cc:1671] finished
0810 18:52:17.323185 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:17.331262 (+ 8077us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1129865,"prepare.queue_time_us":1230378,"prepare.run_cpu_time_us":126,"prepare.run_wall_time_us":126,"raft.queue_time_us":1228044,"raft.run_cpu_time_us":481,"raft.run_wall_time_us":14489,"thread_start_us":220197,"threads_started":1,"wal-append.queue_time_us":576239}
W20260810 18:52:17.374065 16636 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.308s user 0.030s sys 0.069s
I20260810 18:52:17.352572 19929 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:17.377843 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:17.359789 19595 raft_consensus.cc:729] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:17.369027 16748 rpcz_store.cc:276] 0810 18:52:15.808391 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.895791 (+ 87400us) service_pool.cc:227] Handling call
0810 18:52:15.896109 (+ 318us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:16.928754 (+1032645us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:16.928871 (+ 117us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:16.930405 (+ 1534us) log.cc:842] Serialized 26 byte log entry
0810 18:52:17.173059 (+242654us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:17.173194 (+ 135us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:17.173596 (+ 402us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:17.368177 (+194581us) raft_consensus.cc:1671] finished
0810 18:52:17.368202 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:17.368451 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1252554,"raft.run_cpu_time_us":269,"raft.run_wall_time_us":268,"thread_start_us":240109,"threads_started":1,"wal-append.queue_time_us":389075}
I20260810 18:52:17.388298 20437 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:17.389287 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:17.394662 16623 raft_consensus.cc:3037] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:17.395463 19480 leader_election.cc:288] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:17.396204 16450 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.319s user 0.037s sys 0.036s
I20260810 18:52:17.396260 17771 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55586 (request call id 343) took 1800 ms. Trace:
I20260810 18:52:17.396895 20371 raft_consensus.cc:482] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:17.397951 20345 raft_consensus.cc:504] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:17.393218 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:17.400928 16621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1615) took 3889 ms. Trace:
I20260810 18:52:17.401556 17724 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:17.402581 16450 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.418s user 0.038s sys 0.036s
W20260810 18:52:17.403836 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 1.51573e+06us
I20260810 18:52:17.404824 16636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1659) took 1663 ms. Trace:
I20260810 18:52:17.405534 16749 raft_consensus.cc:1261] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:52:17.406270 16749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1728) took 1524 ms. Trace:
I20260810 18:52:17.406545 20134 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:17.405260 16636 rpcz_store.cc:276] 0810 18:52:15.741701 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.946228 (+204527us) service_pool.cc:227] Handling call
0810 18:52:15.946619 (+ 391us) raft_consensus.cc:1457] Updating replica for 2 ops
0810 18:52:15.946996 (+ 377us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:15.947020 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0810 18:52:15.983458 (+ 36438us) log.cc:842] Serialized 52 byte log entry
0810 18:52:17.398940 (+1415482us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:17.400528 (+ 1588us) log.cc:842] Serialized 14 byte log entry
0810 18:52:17.404235 (+ 3707us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:17.404423 (+ 188us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:17.404476 (+ 53us) raft_consensus.cc:1671] finished
0810 18:52:17.404494 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:17.404732 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1333,"raft.queue_time_us":948,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":49871,"thread_start_us":1327129,"threads_started":1,"wal-append.queue_time_us":1325358}
I20260810 18:52:17.408519 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:17.409366 16636 raft_consensus.cc:2454] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:17.409828 16752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 245) took 1529 ms. Trace:
I20260810 18:52:17.410226 16752 rpcz_store.cc:276] 0810 18:52:15.880485 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:17.317166 (+1436681us) service_pool.cc:227] Handling call
0810 18:52:17.384166 (+ 67000us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:17.409410 (+ 25244us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:17.409445 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:17.409503 (+ 58us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:17.409566 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:17.409626 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:17.409753 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:17.402076 17771 rpcz_store.cc:276] 0810 18:52:15.595944 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.598599 (+ 2655us) service_pool.cc:227] Handling call
0810 18:52:15.668506 (+ 69907us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:15.733811 (+ 65305us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:15.735211 (+ 1400us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:15.791976 (+ 56765us) log.cc:842] Serialized 26 byte log entry
0810 18:52:16.240784 (+448808us) raft_consensus.cc:1631] Marking committed up to 6
0810 18:52:17.128028 (+887244us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:17.128235 (+ 207us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:17.395859 (+267624us) raft_consensus.cc:1671] finished
0810 18:52:17.395894 (+ 35us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:17.396096 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5808,"raft.queue_time_us":24649,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":4287,"thread_start_us":427837,"threads_started":1,"wal-append.queue_time_us":1586685}
I20260810 18:52:17.407124 20134 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:17.415042 16496 raft_consensus.cc:1261] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260810 18:52:17.415828 16626 raft_consensus.cc:2454] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:17.417091 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:17.416831 19595 consensus_queue.cc:260] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:17.418221 19595 raft_consensus.cc:3037] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
W20260810 18:52:17.419011 20398 log.cc:924] Time spent T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 1.605s user 0.005s sys 0.016s
I20260810 18:52:17.424870 20345 leader_election.cc:288] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:17.419979 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:17.426903 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:17.428339 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:17.428886 20429 raft_consensus.cc:2726] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:17.429366 16633 raft_consensus.cc:2396] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260810 18:52:17.423038 20159 raft_consensus.cc:3032] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:52:17.429996 20159 raft_consensus.cc:729] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:17.422564 16506 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56468 (request call id 208) took 2470 ms. Trace:
W20260810 18:52:17.421751 16701 leader_election.cc:334] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.626s (SENT)
I20260810 18:52:17.430647 16506 rpcz_store.cc:276] 0810 18:52:14.951708 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.011342 (+ 59634us) service_pool.cc:227] Handling call
0810 18:52:17.231269 (+2219927us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:17.232530 (+ 1261us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:17.422314 (+189784us) spinlock_profiling.cc:243] Waited 2.18 s on lock 0x7b54000e1ad0. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:17.422476 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5020840320}
I20260810 18:52:17.430220 16624 raft_consensus.cc:2454] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:17.432332 16740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1716) took 1870 ms. Trace:
I20260810 18:52:17.433681 19480 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:17.432812 16740 rpcz_store.cc:276] 0810 18:52:15.562268 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.563606 (+ 1338us) service_pool.cc:227] Handling call
0810 18:52:15.564011 (+ 405us) raft_consensus.cc:1457] Updating replica for 3 ops
0810 18:52:15.564473 (+ 462us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:15.564507 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0810 18:52:15.565283 (+ 776us) log.cc:842] Serialized 78 byte log entry
0810 18:52:15.945489 (+380206us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:17.141379 (+1195890us) log.cc:842] Serialized 14 byte log entry
0810 18:52:17.230505 (+ 89126us) log.cc:842] Serialized 14 byte log entry
0810 18:52:17.230837 (+ 332us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:17.230948 (+ 111us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:17.431954 (+201006us) raft_consensus.cc:1671] finished
0810 18:52:17.431980 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:17.432165 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":184106,"prepare.queue_time_us":1199505,"prepare.run_cpu_time_us":113,"prepare.run_wall_time_us":113,"raft.queue_time_us":106574,"raft.run_cpu_time_us":432,"raft.run_wall_time_us":19509,"thread_start_us":339368,"threads_started":1,"wal-append.queue_time_us":1834708}
I20260810 18:52:17.431095 16701 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:17.436430 16501 raft_consensus.cc:1226] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 2. Current term is 3. Ops: []
W20260810 18:52:17.436892 16631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1474) took 2871 ms (client timeout 1650 ms). Trace:
I20260810 18:52:17.434484 19480 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:17.437292 16631 rpcz_store.cc:269] 0810 18:52:14.565610 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:17.436514 (+2870904us) service_pool.cc:214] Skipping call since client already timed out
0810 18:52:17.436654 (+ 140us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260810 18:52:17.431985 20297 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:17.438086 16701 leader_election.cc:334] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.636s (SENT)
I20260810 18:52:17.438613 16701 leader_election.cc:302] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:17.439774 19480 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:17.440058 20297 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:17.440212 16701 leader_election.cc:334] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.686s (SENT)
I20260810 18:52:17.440702 20297 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:17.440671 16701 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:17.441612 19480 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:17.441555 20297 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:17.406687 16749 rpcz_store.cc:276] 0810 18:52:15.881493 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:17.389939 (+1508446us) service_pool.cc:227] Handling call
0810 18:52:17.405005 (+ 15066us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:17.405972 (+ 967us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:17.406202 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:17.442818 19480 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:17.443245 20345 raft_consensus.cc:2726] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:17.443571 20437 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:17.435982 16495 raft_consensus.cc:1226] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:17.445067 20345 raft_consensus.cc:2726] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:17.445998 20437 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:17.446020 20345 raft_consensus.cc:2726] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:17.410243 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:17.420203 20371 raft_consensus.cc:504] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:17.452701 20371 leader_election.cc:288] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:17.387534 20444 raft_consensus.cc:482] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:17.456364 20444 raft_consensus.cc:504] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:17.456612 20159 consensus_queue.cc:260] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:17.457942 20159 raft_consensus.cc:3037] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:17.458043 16750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1693) took 2603 ms. Trace:
I20260810 18:52:17.430261 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:17.458586 16750 rpcz_store.cc:276] 0810 18:52:14.854696 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:14.854829 (+ 133us) service_pool.cc:227] Handling call
0810 18:52:14.855438 (+ 609us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:15.582543 (+727105us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:15.582572 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:15.605748 (+ 23176us) log.cc:842] Serialized 26 byte log entry
0810 18:52:16.107313 (+501565us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:52:16.107430 (+ 117us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:16.107605 (+ 175us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:17.457602 (+1349997us) raft_consensus.cc:1671] finished
0810 18:52:17.457646 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:17.457867 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":54083,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":109,"thread_start_us":1193475,"threads_started":2,"wal-append.queue_time_us":1803807}
I20260810 18:52:17.462947 16747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1696) took 2604 ms. Trace:
I20260810 18:52:17.463302 16628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1656) took 1811 ms. Trace:
I20260810 18:52:17.394021 20298 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:17.504310 20458 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:17.466449 16632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 1631) took 3686 ms (client timeout 1711 ms). Trace:
I20260810 18:52:17.468052 16747 rpcz_store.cc:276] 0810 18:52:14.858312 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:14.871036 (+ 12724us) service_pool.cc:227] Handling call
0810 18:52:17.347663 (+2476627us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:17.367468 (+ 19805us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:17.462719 (+ 95251us) spinlock_profiling.cc:243] Waited 2.48 s on lock 0x7b54002c2ed0. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:17.462891 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5694688640}
W20260810 18:52:17.509057 16632 rpcz_store.cc:269] 0810 18:52:13.779685 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:14.052062 (+272377us) service_pool.cc:227] Handling call
0810 18:52:17.465957 (+3413895us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b54002ff8d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:17.466215 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3393868544}
I20260810 18:52:17.465366 17771 raft_consensus.cc:1226] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 5 LEADER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260810 18:52:17.461941 19480 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:17.428925 20376 leader_election.cc:288] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:17.462410 19914 raft_consensus.cc:482] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
W20260810 18:52:17.508778 20398 log.cc:924] Time spent T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20260810 18:52:17.445123 16621 rpcz_store.cc:276] 0810 18:52:13.511437 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:14.889441 (+1378004us) service_pool.cc:227] Handling call
0810 18:52:14.890027 (+ 586us) raft_consensus.cc:1457] Updating replica for 2 ops
0810 18:52:15.625471 (+735444us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:15.625497 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0810 18:52:15.630286 (+ 4789us) log.cc:842] Serialized 52 byte log entry
0810 18:52:15.830326 (+200040us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:17.138948 (+1308622us) log.cc:842] Serialized 14 byte log entry
0810 18:52:17.322760 (+183812us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:17.345499 (+ 22739us) spinlock_profiling.cc:243] Waited 16.8 ms on lock 0x7b180013d2c0. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1aee0718b1 00007f1aee07e661 00007f1ae51d2638 00007f1ae51d5b11 00007f1ae51d5ac9 00007f1ae51d5a59 00007f1ae51d5a21 00007f1ae51d4d1d 00007f1aee436204 00007f1aee436039 00007f1ae51b4dc6
0810 18:52:17.357254 (+ 11755us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:17.390969 (+ 33715us) raft_consensus.cc:1671] finished
0810 18:52:17.391057 (+ 88us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:17.400865 (+ 9808us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":326094,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":76,"raft.queue_time_us":100615,"raft.run_cpu_time_us":348,"raft.run_wall_time_us":5509,"spinlock_wait_cycles":38582656,"thread_start_us":2326339,"threads_started":4,"wal-append.queue_time_us":501075}
I20260810 18:52:17.543735 16623 raft_consensus.cc:1261] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:52:17.552620 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:17.552851 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:17.553882 16633 raft_consensus.cc:2396] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260810 18:52:17.547282 16636 raft_consensus.cc:2454] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:17.557353 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:17.471661 16628 rpcz_store.cc:276] 0810 18:52:15.652225 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.652401 (+ 176us) service_pool.cc:227] Handling call
0810 18:52:15.652718 (+ 317us) raft_consensus.cc:1457] Updating replica for 2 ops
0810 18:52:15.709182 (+ 56464us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:15.711256 (+ 2074us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0810 18:52:15.723946 (+ 12690us) log.cc:842] Serialized 52 byte log entry
0810 18:52:17.139940 (+1415994us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:17.255580 (+115640us) log.cc:842] Serialized 14 byte log entry
0810 18:52:17.279600 (+ 24020us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:17.279732 (+ 132us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:17.448343 (+168611us) raft_consensus.cc:1671] finished
0810 18:52:17.448360 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:17.463242 (+ 14882us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23368,"prepare.queue_time_us":117069,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":65,"raft.queue_time_us":54756,"raft.run_cpu_time_us":291,"raft.run_wall_time_us":288,"thread_start_us":1392209,"threads_started":1,"wal-append.queue_time_us":1675540}
I20260810 18:52:17.572486 20297 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:17.558034 20345 raft_consensus.cc:482] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:17.589089 16623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1660) took 1825 ms. Trace:
I20260810 18:52:17.589674 16623 rpcz_store.cc:276] 0810 18:52:15.763363 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.954875 (+191512us) service_pool.cc:227] Handling call
0810 18:52:16.033235 (+ 78360us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:17.577498 (+1544263us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:17.589004 (+ 11506us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:17.593248 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:17.594132 16623 raft_consensus.cc:2454] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:52:17.596248 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:17.597174 16621 raft_consensus.cc:2454] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:17.598100 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:17.599169 16573 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.637s user 0.003s sys 0.028s
W20260810 18:52:17.599537 16573 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.691s user 0.003s sys 0.028s
I20260810 18:52:17.589138 20458 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:17.602303 20458 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:17.603003 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:17.603368 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:17.589254 20453 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:17.565137 20298 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:17.548000 20437 consensus_queue.cc:1046] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
I20260810 18:52:17.603677 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:52:17.673844 20425 log.cc:924] Time spent T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
W20260810 18:52:17.612407 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 408972us
I20260810 18:52:17.647616 16626 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:17.613981 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:17.605185 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:17.678501 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:17.678213 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:17.599349 19940 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20260810 18:52:17.677222 16632 raft_consensus.cc:2454] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:17.635910 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:17.714699 16745 raft_consensus.cc:2454] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
I20260810 18:52:17.719415 16743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1708) took 2495 ms. Trace:
W20260810 18:52:17.719776 16701 leader_election.cc:334] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.757s (SENT)
I20260810 18:52:17.720284 16701 leader_election.cc:302] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:17.720679 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 103877us
W20260810 18:52:17.721779 16701 leader_election.cc:334] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.878s (SENT)
I20260810 18:52:17.722234 16701 leader_election.cc:302] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:17.722743 19595 raft_consensus.cc:2726] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 18:52:17.613577 20247 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 3.384s user 0.010s sys 0.023s
W20260810 18:52:17.724952 20247 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 3.605s user 0.012s sys 0.032s
I20260810 18:52:17.725757 16624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1662) took 1949 ms. Trace:
I20260810 18:52:17.727579 16628 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:17.717492 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:17.613986 20439 consensus_queue.cc:1046] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.118s
I20260810 18:52:17.732298 20439 raft_consensus.cc:3032] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:17.732718 20439 raft_consensus.cc:729] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:17.733752 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:17.734134 20444 leader_election.cc:288] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:17.726178 16624 rpcz_store.cc:276] 0810 18:52:15.776211 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:17.432449 (+1656238us) service_pool.cc:227] Handling call
0810 18:52:17.432881 (+ 432us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:17.433397 (+ 516us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:17.433430 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:17.433901 (+ 471us) log.cc:842] Serialized 26 byte log entry
0810 18:52:17.563374 (+129473us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:17.563429 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:17.563536 (+ 107us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:17.725539 (+162003us) raft_consensus.cc:1671] finished
0810 18:52:17.725555 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:17.725674 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1197,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":136,"thread_start_us":128956,"threads_started":1,"wal-append.queue_time_us":205183}
I20260810 18:52:17.732317 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:17.603739 17771 raft_consensus.cc:2454] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:52:17.614249 17775 raft_consensus.cc:1226] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 5 LEADER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:17.742276 20376 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:17.742578 20067 raft_consensus.cc:2726] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:17.735177 16742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1723) took 1961 ms. Trace:
I20260810 18:52:17.747608 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:17.744148 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 102570us
I20260810 18:52:17.745642 16621 raft_consensus.cc:2396] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260810 18:52:17.742300 16623 raft_consensus.cc:2454] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
I20260810 18:52:17.743868 16573 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:52:17.750774 16628 raft_consensus.cc:2454] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:17.748097 16632 raft_consensus.cc:2454] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:17.754019 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:17.755365 16636 raft_consensus.cc:2454] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:17.745221 20124 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:17.760632 20124 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:17.733844 20439 consensus_queue.cc:260] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:17.762243 20439 raft_consensus.cc:3037] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:17.756443 20444 raft_consensus.cc:482] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:17.764822 20444 raft_consensus.cc:504] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:17.767102 20124 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:17.767300 20376 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:17.767980 16505 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1478) took 2005 ms. Trace:
I20260810 18:52:17.768447 16505 rpcz_store.cc:276] 0810 18:52:15.762160 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.763496 (+ 1336us) service_pool.cc:227] Handling call
0810 18:52:15.785810 (+ 22314us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:15.786025 (+ 215us) raft_consensus.cc:1520] Early marking committed up to index 4
0810 18:52:17.267123 (+1481098us) log.cc:842] Serialized 14 byte log entry
0810 18:52:17.718229 (+451106us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:17.739454 (+ 21225us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:17.748909 (+ 9455us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:17.767759 (+ 18850us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:17.767901 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1602904,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":56,"raft.queue_time_us":36477,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":5304,"thread_start_us":1779737,"threads_started":2}
I20260810 18:52:17.769737 20376 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:17.770607 20318 raft_consensus.cc:2781] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260810 18:52:17.771366 20448 log.cc:924] Time spent T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20260810 18:52:17.774290 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:17.639416 20345 raft_consensus.cc:504] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:17.766999 20444 leader_election.cc:288] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:17.775566 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 53846.7us
I20260810 18:52:17.775043 16748 raft_consensus.cc:2454] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:17.780356 17783 raft_consensus.cc:1226] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:17.780620 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:17.783077 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
W20260810 18:52:17.789029 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 45257.2us
I20260810 18:52:17.756404 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:17.778469 17786 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55570 (request call id 438) took 3712 ms (client timeout 1828 ms). Trace:
I20260810 18:52:17.650923 19914 raft_consensus.cc:504] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:17.798763 17786 rpcz_store.cc:269] 0810 18:52:14.066044 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:14.111398 (+ 45354us) service_pool.cc:227] Handling call
0810 18:52:17.778005 (+3666607us) spinlock_profiling.cc:243] Waited 237 ms on lock 0x7b54006a1654. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:17.778225 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":545181568}
W20260810 18:52:17.896334 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 122222us
I20260810 18:52:17.863471 17785 raft_consensus.cc:1226] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:17.896483 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:17.808945 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 60309us
I20260810 18:52:17.800171 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:17.749164 16742 rpcz_store.cc:276] 0810 18:52:15.774088 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.774635 (+ 547us) service_pool.cc:227] Handling call
0810 18:52:15.795559 (+ 20924us) raft_consensus.cc:1457] Updating replica for 2 ops
0810 18:52:15.795921 (+ 362us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:15.795950 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0810 18:52:15.796603 (+ 653us) log.cc:842] Serialized 52 byte log entry
0810 18:52:17.203953 (+1407350us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:17.204486 (+ 533us) log.cc:842] Serialized 14 byte log entry
0810 18:52:17.204797 (+ 311us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:17.204951 (+ 154us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:17.726638 (+521687us) raft_consensus.cc:1671] finished
0810 18:52:17.726661 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:17.735111 (+ 8450us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1719,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":65,"raft.queue_time_us":32331,"raft.run_cpu_time_us":282,"raft.run_wall_time_us":282,"thread_start_us":1406615,"threads_started":1,"wal-append.queue_time_us":1711949}
I20260810 18:52:17.719811 16743 rpcz_store.cc:276] 0810 18:52:15.223590 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.224368 (+ 778us) service_pool.cc:227] Handling call
0810 18:52:15.224721 (+ 353us) raft_consensus.cc:1457] Updating replica for 3 ops
0810 18:52:15.294642 (+ 69921us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:15.294676 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0810 18:52:15.295302 (+ 626us) log.cc:842] Serialized 78 byte log entry
0810 18:52:15.609459 (+314157us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:15.639720 (+ 30261us) log.cc:842] Serialized 14 byte log entry
0810 18:52:15.672460 (+ 32740us) log.cc:842] Serialized 14 byte log entry
0810 18:52:15.706247 (+ 33787us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:15.706329 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:17.418100 (+1711771us) spinlock_profiling.cc:243] Waited 5.9 ms on lock 0x7b44000232cc. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae50e4a95 00007f1ae50e54b2 00007f1ae7d8d511 00007f1ae7d88e1a 00007f1ae7d7cbee 00007f1ae7d7bf36 00007f1ae7d7bb58 00007f1ae7d85551 00007f1ae7d85509 00007f1ae7d85499
0810 18:52:17.663369 (+245269us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b4800251ad8. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1aecfa17f1 00007f1aecf988e7 00007f1aecf9877a 00007f1aed08c95f 00007f1aed08c627 00007f1aed0a0b2c 00007f1aed0a0ad8 00007f1aed0a0a58 00007f1aed0a0a13 00007f1aed09fcbc 00007f1aec021be6
0810 18:52:17.715853 (+ 52484us) raft_consensus.cc:1671] finished
0810 18:52:17.715881 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:17.719185 (+ 3304us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":65800,"prepare.queue_time_us":270052,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":2801,"raft.queue_time_us":47114,"raft.run_cpu_time_us":21996,"raft.run_wall_time_us":2089237,"spinlock_wait_cycles":45343872,"thread_start_us":313709,"threads_started":1,"wal-append.queue_time_us":517842}
I20260810 18:52:17.900225 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:17.835985 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:17.836320 16632 raft_consensus.cc:2454] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:17.907264 16508 raft_consensus.cc:2454] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
I20260810 18:52:17.907640 16626 raft_consensus.cc:2454] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:17.866936 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:17.909521 19914 leader_election.cc:288] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:17.910547 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:17.910841 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:17.869024 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:17.788276 20444 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:17.793953 17726 connection.cc:232] RPC call timeout handler was delayed by -1.0417519151233137s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1625s for a response
I20260810 18:52:17.829754 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:17.907651 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:17.919435 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:17.838204 16701 leader_election.cc:334] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.041s (SENT)
I20260810 18:52:17.928280 16632 raft_consensus.cc:2454] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:17.779575 20345 leader_election.cc:288] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:17.922708 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:17.930786 16495 raft_consensus.cc:2454] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 4.
I20260810 18:52:17.807014 16633 raft_consensus.cc:2454] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:17.932719 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:17.933858 16633 raft_consensus.cc:2454] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
I20260810 18:52:17.935384 16623 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:17.937889 16701 leader_election.cc:302] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:17.902156 20464 raft_consensus.cc:2781] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:17.945998 20464 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:17.946389 20464 raft_consensus.cc:3037] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:17.943385 17770 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55586 (request call id 345) took 1986 ms. Trace:
I20260810 18:52:17.759001 20453 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:17.940804 20305 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:17.944015 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:17.944069 16628 raft_consensus.cc:2454] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:17.957916 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:17.775766 19480 consensus_queue.cc:1035] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.152s
I20260810 18:52:17.835103 20037 raft_consensus.cc:482] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:17.962529 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:17.953832 16506 raft_consensus.cc:2454] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 4.
I20260810 18:52:17.962839 20345 raft_consensus.cc:2726] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:17.966949 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:17.967870 16632 raft_consensus.cc:2454] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:17.971406 20453 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:17.971691 20298 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:17.973306 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 165285us
W20260810 18:52:17.984923 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 161251us
I20260810 18:52:17.984596 20305 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:17.954733 17770 rpcz_store.cc:276] 0810 18:52:15.851542 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.859539 (+ 7997us) service_pool.cc:227] Handling call
0810 18:52:17.365514 (+1505975us) spinlock_profiling.cc:243] Waited 5.33 ms on lock 0x7b5800005498. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed0502b0 00007f1aed03b7f9 00007f1aed03b2a2 00007f1ae6721684
0810 18:52:17.365979 (+ 465us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:17.366733 (+ 754us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:17.366784 (+ 51us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:17.386942 (+ 20158us) log.cc:842] Serialized 26 byte log entry
0810 18:52:17.506416 (+119474us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:17.508851 (+ 2435us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:17.518705 (+ 9854us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:17.785871 (+267166us) raft_consensus.cc:1671] finished
0810 18:52:17.788278 (+ 2407us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:17.829808 (+ 41530us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":61749,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":153,"spinlock_wait_cycles":12257536,"thread_start_us":73735,"threads_started":1,"wal-append.queue_time_us":183008}
I20260810 18:52:17.988816 16628 raft_consensus.cc:1226] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:18.003592 16628 raft_consensus.cc:1226] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260810 18:52:18.006910 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 168835us
I20260810 18:52:17.941643 20444 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:17.986955 20305 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:18.026639 16624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1668) took 2193 ms. Trace:
W20260810 18:52:18.024916 16507 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56468 (request call id 207) took 2999 ms (client timeout 2189 ms). Trace:
I20260810 18:52:18.027150 16624 rpcz_store.cc:276] 0810 18:52:15.828472 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:17.937642 (+2109170us) service_pool.cc:227] Handling call
0810 18:52:17.973807 (+ 36165us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:18.002062 (+ 28255us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:18.002100 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:18.007786 (+ 5686us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:18.021371 (+ 13585us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:18.021452 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:18.021583 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 18:52:18.012056 16701 leader_election.cc:334] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.138s (SENT)
I20260810 18:52:18.034078 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:18.029803 16701 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:18.027223 16507 rpcz_store.cc:269] 0810 18:52:14.951389 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:14.959128 (+ 7739us) service_pool.cc:227] Handling call
0810 18:52:17.950999 (+2991871us) spinlock_profiling.cc:243] Waited 2.35 s on lock 0x7b54001cbbd4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:17.951208 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5415366400}
I20260810 18:52:18.048959 20458 consensus_queue.cc:1046] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260810 18:52:18.045800 17727 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:18.059823 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:18.064503 16632 raft_consensus.cc:2454] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:18.065414 16695 leader_election.cc:302] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:18.066329 20297 raft_consensus.cc:2781] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:18.069288 16631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1663) took 2289 ms. Trace:
I20260810 18:52:18.076485 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:18.049892 20458 consensus_queue.cc:1225] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer 831925dd8b644ebab7eb0d672a94eddd log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260810 18:52:18.078510 16447 leader_election.cc:302] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:18.061864 20453 raft_consensus.cc:2726] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:18.082738 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:18.079469 16631 rpcz_store.cc:276] 0810 18:52:15.780314 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:17.438150 (+1657836us) service_pool.cc:227] Handling call
0810 18:52:17.438587 (+ 437us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:17.438986 (+ 399us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:17.439013 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:17.567752 (+128739us) log.cc:842] Serialized 26 byte log entry
0810 18:52:17.739145 (+171393us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:17.739403 (+ 258us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:17.739485 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:18.069014 (+329529us) raft_consensus.cc:1671] finished
0810 18:52:18.069039 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:18.069196 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4781,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":147,"thread_start_us":103792,"threads_started":1,"wal-append.queue_time_us":461805}
I20260810 18:52:18.084865 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:18.079399 20297 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:18.105847 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:18.050026 20037 raft_consensus.cc:504] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:18.106278 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:18.106487 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:18.113385 16628 raft_consensus.cc:2454] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:18.113525 16749 raft_consensus.cc:2454] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 4.
I20260810 18:52:18.106962 20297 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:18.120199 16624 raft_consensus.cc:2354] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3.
I20260810 18:52:18.095880 20453 raft_consensus.cc:3037] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
W20260810 18:52:18.122956 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 119101us
I20260810 18:52:18.119939 17727 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:18.125660 19480 raft_consensus.cc:2726] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260810 18:52:18.128078 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:18.143690 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 111333us
W20260810 18:52:18.146154 16701 leader_election.cc:334] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.196s (SENT)
I20260810 18:52:18.143177 20480 consensus_queue.cc:1035] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.062s
W20260810 18:52:18.152340 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 228001us
I20260810 18:52:18.153474 16701 leader_election.cc:302] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:18.156092 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:18.158196 16624 raft_consensus.cc:2396] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260810 18:52:18.147567 20305 raft_consensus.cc:2781] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:18.223459 20373 raft_consensus.cc:3032] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 4 LEADER]: Stepping down as leader of term 4
I20260810 18:52:18.224083 20373 raft_consensus.cc:729] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:18.151007 20464 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:18.225298 20373 consensus_queue.cc:260] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:18.207981 20305 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:18.249259 16632 raft_consensus.cc:1261] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260810 18:52:18.229888 20345 raft_consensus.cc:2726] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260810 18:52:18.248055 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 218401us
I20260810 18:52:18.257243 16631 raft_consensus.cc:1226] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 5. Ops: []
I20260810 18:52:18.241293 20444 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:18.247291 20305 raft_consensus.cc:3037] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Advancing to term 5
W20260810 18:52:18.284770 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 138692us
I20260810 18:52:18.267257 20373 raft_consensus.cc:3037] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:18.260525 20436 consensus_queue.cc:1035] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.025s
I20260810 18:52:18.292801 16621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1667) took 2464 ms. Trace:
I20260810 18:52:18.342303 16447 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:18.289243 17770 raft_consensus.cc:1226] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 5 LEADER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 4. Current term is 5. Ops: []
W20260810 18:52:18.350991 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 60390.8us
W20260810 18:52:18.355878 20485 log.cc:924] Time spent T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.093s user 0.000s sys 0.001s
W20260810 18:52:18.357417 20486 log.cc:924] Time spent T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20260810 18:52:18.358887 16626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1664) took 2538 ms. Trace:
I20260810 18:52:18.359308 16626 rpcz_store.cc:276] 0810 18:52:15.820020 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:17.912446 (+2092426us) service_pool.cc:227] Handling call
0810 18:52:17.912826 (+ 380us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:17.913289 (+ 463us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:17.913317 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:17.913791 (+ 474us) log.cc:842] Serialized 26 byte log entry
0810 18:52:18.026048 (+112257us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:52:18.026105 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:18.026199 (+ 94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:18.358656 (+332457us) raft_consensus.cc:1671] finished
0810 18:52:18.358676 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:18.358824 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":120,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":138,"thread_start_us":105949,"threads_started":1,"wal-append.queue_time_us":298978}
I20260810 18:52:18.353564 16450 leader_election.cc:302] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:18.355930 20318 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:18.373219 20318 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:18.370807 17780 raft_consensus.cc:1226] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 5 LEADER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:18.391637 16742 raft_consensus.cc:1261] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260810 18:52:18.369987 20037 leader_election.cc:288] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:18.389221 20436 raft_consensus.cc:2726] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260810 18:52:18.399050 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:18.407415 19996 raft_consensus.cc:2781] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:18.407886 19996 raft_consensus.cc:482] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:18.408224 19996 raft_consensus.cc:3037] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:18.383813 20255 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:18.412845 20255 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:18.417678 16752 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 5.
W20260810 18:52:18.418598 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 128283us
I20260810 18:52:18.421490 20329 consensus_queue.cc:1035] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.020s
I20260810 18:52:18.423671 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:18.424299 17770 raft_consensus.cc:3037] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:18.393242 20464 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:18.430151 16633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1665) took 2602 ms. Trace:
I20260810 18:52:18.430666 16633 rpcz_store.cc:276] 0810 18:52:15.827618 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:17.934733 (+2107115us) service_pool.cc:227] Handling call
0810 18:52:17.982066 (+ 47333us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:18.052725 (+ 70659us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:18.052765 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:18.069460 (+ 16695us) log.cc:842] Serialized 26 byte log entry
0810 18:52:18.261505 (+192045us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:18.261556 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:18.261643 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:18.429865 (+168222us) raft_consensus.cc:1671] finished
0810 18:52:18.429892 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:18.430078 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":20708,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":105,"thread_start_us":142170,"threads_started":1,"wal-append.queue_time_us":333441}
I20260810 18:52:18.432909 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:18.449117 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 168793us
W20260810 18:52:18.493705 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 56903.3us
W20260810 18:52:18.519953 20508 log.cc:924] Time spent T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.126s user 0.000s sys 0.000s
I20260810 18:52:18.531795 20255 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:18.550429 16623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1666) took 2722 ms. Trace:
I20260810 18:52:18.550942 16623 rpcz_store.cc:276] 0810 18:52:15.827887 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:17.954971 (+2127084us) service_pool.cc:227] Handling call
0810 18:52:18.006551 (+ 51580us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:18.069616 (+ 63065us) raft_consensus.cc:1520] Early marking committed up to index 3
0810 18:52:18.265788 (+196172us) log.cc:842] Serialized 14 byte log entry
0810 18:52:18.549980 (+284192us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:18.550050 (+ 70us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:18.550113 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:18.550203 (+ 90us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:18.550358 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":66558,"prepare.queue_time_us":294638,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"thread_start_us":376761,"threads_started":2}
I20260810 18:52:18.547668 19786 raft_consensus.cc:3037] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:18.566504 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:52:18.585358 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 225572us
I20260810 18:52:18.590838 17785 raft_consensus.cc:2454] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:18.593865 16695 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:18.599030 16740 raft_consensus.cc:1261] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260810 18:52:18.578058 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:18.604809 16507 raft_consensus.cc:1226] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260810 18:52:18.605093 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:18.606094 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:18.608619 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:18.611716 16495 raft_consensus.cc:1226] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:18.613607 17785 raft_consensus.cc:2379] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 6: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:52:18.610986 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:18.616112 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:18.624750 17724 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:18.627988 16740 raft_consensus.cc:2454] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
W20260810 18:52:18.618808 17778 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.272s user 0.009s sys 0.008s
I20260810 18:52:18.631428 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:18.632651 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:18.625753 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:18.634868 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:18.632982 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:18.632228 17778 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.392s user 0.009s sys 0.008s
I20260810 18:52:18.634889 17775 raft_consensus.cc:1261] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260810 18:52:18.673864 17786 raft_consensus.cc:3032] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 4 LEADER]: Stepping down as leader of term 4
I20260810 18:52:18.678268 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:18.634835 20329 raft_consensus.cc:2781] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260810 18:52:18.687726 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 87374.7us
I20260810 18:52:18.689072 17786 raft_consensus.cc:729] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
W20260810 18:52:18.696344 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 102590us
I20260810 18:52:18.712203 16745 raft_consensus.cc:1261] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260810 18:52:18.694201 20329 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
W20260810 18:52:18.731943 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 170795us
I20260810 18:52:18.693889 20318 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:18.745859 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:18.753348 16505 raft_consensus.cc:2454] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:18.722529 20329 raft_consensus.cc:3037] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:18.763214 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:18.762669 17786 consensus_queue.cc:260] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:18.731812 20305 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:18.809729 16624 raft_consensus.cc:1261] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260810 18:52:18.768095 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:18.825099 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:18.800293 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:18.854389 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 126720us
I20260810 18:52:18.824115 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:18.824592 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:18.933161 17786 raft_consensus.cc:3037] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:18.825742 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:18.825453 20527 log.cc:924] Time spent T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.147s user 0.001s sys 0.000s
I20260810 18:52:18.824177 20513 raft_consensus.cc:2781] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260810 18:52:19.007748 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 82529.2us
I20260810 18:52:18.824594 19863 raft_consensus.cc:482] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:18.765381 17770 raft_consensus.cc:2454] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:19.023518 16745 raft_consensus.cc:2396] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 7 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260810 18:52:19.030926 16701 leader_election.cc:334] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.829s (SENT)
I20260810 18:52:18.810019 19996 raft_consensus.cc:504] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:19.016085 20513 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:19.086661 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:19.089043 16695 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:19.092648 16573 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:52:19.083014 17780 raft_consensus.cc:2379] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 5: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:52:18.776213 20538 consensus_queue.cc:1035] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.302s
I20260810 18:52:19.096285 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:19.090058 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:19.090925 20453 consensus_meta.cc:287] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.959s user 0.012s sys 0.014s
W20260810 18:52:19.091620 19786 consensus_meta.cc:287] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.514s user 0.009s sys 0.001s
I20260810 18:52:19.088359 20513 raft_consensus.cc:3037] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
W20260810 18:52:19.120970 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 99220us
I20260810 18:52:19.032977 20530 consensus_queue.cc:1035] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.084s
W20260810 18:52:19.116747 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 102377us
I20260810 18:52:19.108510 20537 raft_consensus.cc:2781] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:19.116063 20061 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:19.113008 20453 raft_consensus.cc:2726] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:19.115046 19858 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:19.098050 20436 consensus_queue.cc:1035] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260810 18:52:19.082636 19863 raft_consensus.cc:504] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:19.132309 16450 leader_election.cc:302] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:19.134617 16747 raft_consensus.cc:1226] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 4. Current term is 5. Ops: []
W20260810 18:52:19.135917 16502 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52352 (request call id 1475) took 4177 ms (client timeout 3080 ms). Trace:
I20260810 18:52:19.136286 19996 leader_election.cc:288] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:19.133960 19858 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:19.136415 16502 rpcz_store.cc:269] 0810 18:52:14.922335 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:14.977183 (+ 54848us) service_pool.cc:227] Handling call
0810 18:52:19.061983 (+4084800us) spinlock_profiling.cc:243] Waited 439 ms on lock 0x7b5400116054. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:19.099916 (+ 37933us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1009198976}
I20260810 18:52:19.137718 17787 raft_consensus.cc:1261] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260810 18:52:19.123569 20318 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:19.144879 20400 log.cc:924] Time spent T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 3.335s user 0.016s sys 0.007s
I20260810 18:52:19.147961 16626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1693) took 1016 ms. Trace:
W20260810 18:52:19.149080 20416 log.cc:924] Time spent T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 1.751s user 0.008s sys 0.029s
W20260810 18:52:19.149578 20533 log.cc:924] Time spent T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
W20260810 18:52:19.150842 20553 log.cc:924] Time spent T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260810 18:52:19.151942 16622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1657) took 3445 ms. Trace:
I20260810 18:52:19.152210 16628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1690) took 1241 ms. Trace:
W20260810 18:52:19.153129 20546 log.cc:924] Time spent T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.056s user 0.000s sys 0.001s
I20260810 18:52:19.156782 16450 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:19.152627 16628 rpcz_store.cc:276] 0810 18:52:17.911159 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:18.604369 (+693210us) service_pool.cc:227] Handling call
0810 18:52:18.607835 (+ 3466us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:18.626783 (+ 18948us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:18.626830 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:18.639564 (+ 12734us) log.cc:842] Serialized 26 byte log entry
0810 18:52:18.748307 (+108743us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:18.748358 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:18.748449 (+ 91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:19.152001 (+403552us) raft_consensus.cc:1671] finished
0810 18:52:19.152021 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:19.152150 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":24961,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":21233,"thread_start_us":43359,"threads_started":1,"wal-append.queue_time_us":126851}
I20260810 18:52:19.148363 16626 rpcz_store.cc:276] 0810 18:52:18.131800 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:18.650245 (+518445us) service_pool.cc:227] Handling call
0810 18:52:18.650600 (+ 355us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:18.650975 (+ 375us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:18.651002 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:18.651465 (+ 463us) log.cc:842] Serialized 26 byte log entry
0810 18:52:18.806997 (+155532us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:18.807052 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:18.807166 (+ 114us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:19.147754 (+340588us) raft_consensus.cc:1671] finished
0810 18:52:19.147773 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:19.147902 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":25499,"spinlock_wait_cycles":26289536,"thread_start_us":155045,"threads_started":1,"wal-append.queue_time_us":432719}
I20260810 18:52:19.131409 20537 raft_consensus.cc:686] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
W20260810 18:52:19.167420 20528 log.cc:924] Time spent T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.374s user 0.000s sys 0.001s
W20260810 18:52:19.169154 20555 log.cc:924] Time spent T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.109s user 0.000s sys 0.001s
I20260810 18:52:19.152325 16622 rpcz_store.cc:276] 0810 18:52:15.706627 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.706776 (+ 149us) service_pool.cc:227] Handling call
0810 18:52:15.707113 (+ 337us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:15.716304 (+ 9191us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:15.716349 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:15.716862 (+ 513us) log.cc:842] Serialized 26 byte log entry
0810 18:52:17.137508 (+1420646us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:52:17.137615 (+ 107us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:17.137775 (+ 160us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:19.054412 (+1916637us) spinlock_profiling.cc:243] Waited 1.79 ms on lock 0x7b44000232cc. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae50e4a95 00007f1ae50e54b2 00007f1ae7d8d511 00007f1ae7d88e1a 00007f1ae7d7cbee 00007f1ae7d7bf36 00007f1ae7d7bb58 00007f1ae7d85551 00007f1ae7d85509 00007f1ae7d85499
0810 18:52:19.151579 (+ 97167us) raft_consensus.cc:1671] finished
0810 18:52:19.151605 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:19.151845 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8794,"raft.queue_time_us":9628,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":3675,"spinlock_wait_cycles":4127104,"thread_start_us":1404326,"threads_started":1,"wal-append.queue_time_us":1631618}
I20260810 18:52:19.172909 20580 raft_consensus.cc:2781] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260810 18:52:19.166477 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 58470.2us
I20260810 18:52:19.167443 20134 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:19.162180 16701 leader_election.cc:334] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 3.022s (SENT)
I20260810 18:52:19.234949 17786 raft_consensus.cc:1261] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260810 18:52:19.242704 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 65056.8us
I20260810 18:52:19.134806 20061 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:19.245958 20558 raft_consensus.cc:2781] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:19.246487 20558 raft_consensus.cc:482] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:19.246862 20558 raft_consensus.cc:3037] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:19.247269 20565 raft_consensus.cc:2726] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:19.247406 20061 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:19.248736 20306 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:19.249248 20306 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:19.249814 16701 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:19.250566 19786 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:19.251187 16746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 240) took 3938 ms. Trace:
I20260810 18:52:19.251113 19786 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:19.251842 20306 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:19.253105 19786 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:19.175601 20305 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:19.254194 16631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1676) took 1533 ms. Trace:
I20260810 18:52:19.253000 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:19.270309 16631 rpcz_store.cc:276] 0810 18:52:17.721138 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:18.129339 (+408201us) service_pool.cc:227] Handling call
0810 18:52:18.129707 (+ 368us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:18.261446 (+131739us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:19.253956 (+992510us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b540035d754. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:19.254133 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":264381184}
W20260810 18:52:19.253544 20541 log.cc:924] Time spent T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.371s user 0.000s sys 0.001s
W20260810 18:52:19.308910 16752 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.585s user 0.008s sys 0.028s
W20260810 18:52:19.309417 16752 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.586s user 0.008s sys 0.028s
I20260810 18:52:19.314329 16636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1516) took 1588 ms. Trace:
I20260810 18:52:19.244364 20580 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
W20260810 18:52:19.380609 20329 consensus_meta.cc:287] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.603s user 0.009s sys 0.000s
I20260810 18:52:19.403167 16450 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:19.380550 20537 consensus_queue.cc:237] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:19.403883 20329 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:19.404892 19944 raft_consensus.cc:2726] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:19.372537 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:19.405928 16504 raft_consensus.cc:2454] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:52:19.379145 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:19.383204 20566 raft_consensus.cc:2726] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:19.407320 20580 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:19.251576 16746 rpcz_store.cc:276] 0810 18:52:15.313015 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.325753 (+ 12738us) service_pool.cc:227] Handling call
0810 18:52:15.326359 (+ 606us) raft_consensus.cc:1457] Updating replica for 3 ops
0810 18:52:15.358318 (+ 31959us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:15.358364 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0810 18:52:15.359209 (+ 845us) log.cc:842] Serialized 78 byte log entry
0810 18:52:15.628026 (+268817us) raft_consensus.cc:1631] Marking committed up to 6
0810 18:52:15.671359 (+ 43333us) log.cc:842] Serialized 14 byte log entry
0810 18:52:15.710217 (+ 38858us) log.cc:842] Serialized 14 byte log entry
0810 18:52:15.718719 (+ 8502us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:15.718790 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:15.807175 (+ 88385us) spinlock_profiling.cc:243] Waited 1.41 ms on lock 0x7b180001e3a0. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1aee0718b1 00007f1aee07e661 00007f1ae51d2638 00007f1ae51d5b11 00007f1ae51d5ac9 00007f1ae51d5a59 00007f1ae51d5a21 00007f1ae51d4d1d 00007f1aee436204 00007f1aee436039 00007f1ae51b4dc6
0810 18:52:19.090012 (+3282837us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b44000232cc. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae50e4a95 00007f1ae50e54b2 00007f1ae7d8d511 00007f1ae7d88e1a 00007f1ae7d7cbee 00007f1ae7d7bf36 00007f1ae7d7bb58 00007f1ae7d85551 00007f1ae7d85509 00007f1ae7d85499
0810 18:52:19.160205 (+ 70193us) raft_consensus.cc:1671] finished
0810 18:52:19.160228 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:19.250981 (+ 90753us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":96078,"prepare.queue_time_us":209368,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":55202,"raft.run_cpu_time_us":353,"raft.run_wall_time_us":63101,"spinlock_wait_cycles":29033600,"thread_start_us":208365,"threads_started":1,"wal-append.queue_time_us":388260}
I20260810 18:52:19.358222 16633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1687) took 1449 ms. Trace:
I20260810 18:52:19.378530 19907 consensus_queue.cc:1035] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.114s
I20260810 18:52:19.411582 16633 rpcz_store.cc:276] 0810 18:52:17.908727 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:18.437013 (+528286us) service_pool.cc:227] Handling call
0810 18:52:18.437355 (+ 342us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:18.437629 (+ 274us) raft_consensus.cc:1520] Early marking committed up to index 3
0810 18:52:18.706604 (+268975us) log.cc:842] Serialized 14 byte log entry
0810 18:52:19.357780 (+651176us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:19.357853 (+ 73us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:19.357909 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:19.357992 (+ 83us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:19.358150 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34876,"prepare.queue_time_us":118806,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":53,"raft.queue_time_us":15828,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":6196,"thread_start_us":789708,"threads_started":3,"wal-append.queue_time_us":434814}
W20260810 18:52:19.412771 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 279100us
I20260810 18:52:19.413746 20538 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:19.414319 20538 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:19.415562 16636 rpcz_store.cc:276] 0810 18:52:17.726163 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:19.252692 (+1526529us) service_pool.cc:227] Handling call
0810 18:52:19.314272 (+ 61580us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:19.416509 20538 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:19.418678 20584 raft_consensus.cc:482] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:19.419258 20584 raft_consensus.cc:504] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:19.316318 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:19.421370 20584 leader_election.cc:288] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:19.426808 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:19.378643 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:19.431182 20444 raft_consensus.cc:482] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:19.431588 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:19.431756 20444 raft_consensus.cc:504] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:19.432333 16502 raft_consensus.cc:2379] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 7: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:19.433786 20444 leader_election.cc:288] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:19.406839 20297 consensus_meta.cc:287] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 1.264s user 0.013s sys 0.000s
W20260810 18:52:19.437287 16701 leader_election.cc:334] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.510s (SENT)
I20260810 18:52:19.449754 16701 leader_election.cc:302] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
W20260810 18:52:19.398173 20424 log.cc:924] Time spent T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 2.034s user 0.009s sys 0.016s
W20260810 18:52:19.393537 20569 log.cc:924] Time spent T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
W20260810 18:52:19.399199 20568 log.cc:924] Time spent T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.053s user 0.000s sys 0.001s
I20260810 18:52:19.454756 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:19.459864 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:19.470258 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:19.463366 16740 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.697s user 0.014s sys 0.038s
W20260810 18:52:19.472601 16740 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.707s user 0.014s sys 0.038s
I20260810 18:52:19.481005 17783 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55602 (request call id 351) took 3924 ms. Trace:
I20260810 18:52:19.481431 17783 rpcz_store.cc:276] 0810 18:52:15.556267 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.556422 (+ 155us) service_pool.cc:227] Handling call
0810 18:52:17.779974 (+2223552us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:17.808196 (+ 28222us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:19.469424 (+1661228us) spinlock_profiling.cc:243] Waited 2.22 s on lock 0x7b54006a1650. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:19.480922 (+ 11498us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5112755072}
I20260810 18:52:19.486864 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:19.487706 16748 raft_consensus.cc:3037] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:19.491631 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:19.512328 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:19.516327 20076 raft_consensus.cc:3032] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 4 LEADER]: Stepping down as leader of term 4
I20260810 18:52:19.516770 20076 raft_consensus.cc:729] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:19.517704 20076 consensus_queue.cc:260] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:19.518441 20035 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260810 18:52:19.518867 20076 raft_consensus.cc:3037] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:19.463201 17770 raft_consensus.cc:1226] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:19.521551 17724 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:19.510057 20405 log.cc:924] Time spent T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 3.123s user 0.001s sys 0.007s
I20260810 18:52:19.508944 20580 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:19.510797 16573 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:19.492676 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:19.463922 20566 raft_consensus.cc:2726] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:19.503152 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
W20260810 18:52:19.530543 16701 leader_election.cc:334] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.532s (SENT)
I20260810 18:52:19.539296 16502 raft_consensus.cc:1261] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260810 18:52:19.455216 20584 consensus_queue.cc:1035] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.046s
I20260810 18:52:19.541383 16701 leader_election.cc:302] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:19.453763 20297 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:19.547187 16750 raft_consensus.cc:1226] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:19.527587 20035 raft_consensus.cc:2741] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260810 18:52:19.548049 16495 raft_consensus.cc:1226] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:19.556474 16502 raft_consensus.cc:1226] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260810 18:52:19.556300 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 91858.8us
I20260810 18:52:19.500342 20306 consensus_queue.cc:1046] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.054s
I20260810 18:52:19.546907 20494 raft_consensus.cc:2726] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:19.541929 20436 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:19.538249 20558 raft_consensus.cc:504] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:19.560670 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:19.577023 20566 raft_consensus.cc:2726] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260810 18:52:19.592664 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 68075.9us
W20260810 18:52:19.596451 16701 leader_election.cc:334] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.541s (SENT)
I20260810 18:52:19.597637 16447 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:19.600273 16496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56468 (request call id 205) took 4653 ms. Trace:
I20260810 18:52:19.600812 16701 leader_election.cc:302] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:19.601622 16496 rpcz_store.cc:276] 0810 18:52:14.946657 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.010238 (+ 63581us) service_pool.cc:227] Handling call
0810 18:52:15.032304 (+ 22066us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:17.415497 (+2383193us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:19.595076 (+2179579us) spinlock_profiling.cc:243] Waited 2.38 s on lock 0x7b54001374d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:19.600208 (+ 5132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5478168448}
W20260810 18:52:19.608187 16701 leader_election.cc:334] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.650s (SENT)
I20260810 18:52:19.593809 20436 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:19.616961 20436 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:19.602630 20566 raft_consensus.cc:2726] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:19.608754 16701 leader_election.cc:302] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:19.635547 20580 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:19.641405 20513 consensus_meta.cc:287] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.513s user 0.010s sys 0.001s
I20260810 18:52:19.642256 20558 leader_election.cc:288] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:19.643162 20592 raft_consensus.cc:2726] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:19.641947 20513 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:19.644470 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:19.626466 20530 raft_consensus.cc:3037] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:19.654521 16745 raft_consensus.cc:1226] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260810 18:52:19.656843 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:19.669772 16324 catalog_manager.cc:4841] TS 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): tablet cdfc652dd163415b94d3f0bc3931a07f (table test-workload [id=5aa4952cbee8447ca410249e65af5af6]) successfully deleted
I20260810 18:52:19.677069 20297 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:19.666906 16748 raft_consensus.cc:2454] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:18.322695 16621 rpcz_store.cc:276] 0810 18:52:15.828198 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:17.941892 (+2113694us) service_pool.cc:227] Handling call
0810 18:52:17.993446 (+ 51554us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:18.070563 (+ 77117us) raft_consensus.cc:1520] Early marking committed up to index 6
0810 18:52:18.225469 (+154906us) log.cc:842] Serialized 14 byte log entry
0810 18:52:18.292047 (+ 66578us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:18.292201 (+ 154us) raft_consensus.cc:1631] Marking committed up to 6
0810 18:52:18.292322 (+ 121us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:18.292448 (+ 126us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:18.292676 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":215772,"raft.queue_time_us":4307,"spinlock_wait_cycles":56418688,"thread_start_us":202731,"threads_started":2}
W20260810 18:52:19.683272 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 75121.2us
W20260810 18:52:19.683974 16701 leader_election.cc:334] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.672s (SENT)
I20260810 18:52:19.684316 16701 leader_election.cc:302] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:19.685190 20493 log.cc:924] Time spent T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 1.306s user 0.004s sys 0.000s
W20260810 18:52:19.685333 16701 leader_election.cc:334] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.722s (SENT)
I20260810 18:52:19.685933 20580 raft_consensus.cc:2726] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:19.686712 16701 leader_election.cc:302] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:19.691778 20306 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:19.695879 16496 raft_consensus.cc:1226] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:19.695551 20306 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:19.704947 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:19.708280 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:19.707986 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:19.714946 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:19.699091 20306 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:19.739470 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 141919us
W20260810 18:52:19.759975 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 92573.3us
I20260810 18:52:19.765015 20329 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:19.766371 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
W20260810 18:52:19.904294 17726 leader_election.cc:334] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.625s (SENT)
I20260810 18:52:19.904785 16501 raft_consensus.cc:3037] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:19.772948 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:19.774955 16573 leader_election.cc:334] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Timed out: RequestConsensusVote RPC to 127.14.32.193:40791 timed out after 1.589s (SENT)
I20260810 18:52:19.905516 16573 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:19.798096 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:19.798696 20591 raft_consensus.cc:3032] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:19.912442 20591 raft_consensus.cc:729] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:19.805691 20061 consensus_queue.cc:1046] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260810 18:52:19.807894 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:19.808874 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:19.809077 19863 leader_election.cc:288] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:19.814723 16505 raft_consensus.cc:1226] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:19.813709 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:19.826567 16504 raft_consensus.cc:1226] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:19.832523 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:19.924121 16495 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:19.793557 20530 raft_consensus.cc:2726] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:19.702538 20592 raft_consensus.cc:2726] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260810 18:52:19.901539 16701 leader_election.cc:334] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.835s (SENT)
I20260810 18:52:19.944489 16701 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:19.768055 20608 log.cc:924] Time spent T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.081s user 0.000s sys 0.001s
I20260810 18:52:19.907019 16447 leader_election.cc:302] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:19.911463 20458 raft_consensus.cc:2726] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:19.915293 16749 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:19.913460 20591 consensus_queue.cc:260] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:19.920680 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 145760us
I20260810 18:52:19.921152 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:19.921797 20513 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:19.928165 20437 raft_consensus.cc:3032] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:19.930382 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:19.932725 16747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 255) took 1789 ms. Trace:
I20260810 18:52:19.954607 16747 rpcz_store.cc:276] 0810 18:52:18.142851 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:18.143007 (+ 156us) service_pool.cc:227] Handling call
0810 18:52:18.147318 (+ 4311us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:19.140609 (+993291us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:19.932517 (+791908us) spinlock_profiling.cc:243] Waited 986 ms on lock 0x7b54002ee5d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:19.932671 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2267968896}
I20260810 18:52:19.935424 16508 raft_consensus.cc:1226] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:19.936682 17787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55602 (request call id 352) took 4358 ms. Trace:
I20260810 18:52:19.956164 17787 rpcz_store.cc:276] 0810 18:52:15.578023 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:17.725110 (+2147087us) service_pool.cc:227] Handling call
0810 18:52:17.725501 (+ 391us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:19.138106 (+1412605us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:19.936406 (+798300us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b54006a04d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:19.936612 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3246605568}
I20260810 18:52:19.937837 16495 raft_consensus.cc:2454] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:19.941722 20540 consensus_queue.cc:1046] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260810 18:52:19.950657 20591 raft_consensus.cc:3037] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:19.947453 16623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1689) took 2037 ms. Trace:
W20260810 18:52:19.950350 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 48857.8us
I20260810 18:52:19.962360 17783 raft_consensus.cc:1226] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 4. Current term is 5. Ops: []
W20260810 18:52:19.949921 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 241279us
I20260810 18:52:19.944536 16501 raft_consensus.cc:2454] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:19.948318 20439 raft_consensus.cc:482] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:19.960470 16623 rpcz_store.cc:276] 0810 18:52:17.910434 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:18.552708 (+642274us) service_pool.cc:227] Handling call
0810 18:52:18.633483 (+ 80775us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:18.633677 (+ 194us) raft_consensus.cc:1520] Early marking committed up to index 3
0810 18:52:18.635864 (+ 2187us) log.cc:842] Serialized 14 byte log entry
0810 18:52:19.816409 (+1180545us) spinlock_profiling.cc:243] Waited 3.99 ms on lock 0x7b1800284920. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1aee0718b1 00007f1aee07e661 00007f1ae51b4345 00007f1aedfeba39 00007f1ae51d18bb 00007f1ae51cfcbf 00007f1ae51d1e22 00007f1ae7d7b5bb 00007f1ae7d81063 00007f1ae7d8157b 00007f1ae810488a
0810 18:52:19.947018 (+130609us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:19.947101 (+ 83us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:19.947162 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:19.947246 (+ 84us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:19.947378 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15901,"prepare.queue_time_us":1142,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":15918,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":1591,"spinlock_wait_cycles":9202944,"thread_start_us":27462,"threads_started":1,"wal-append.queue_time_us":1266276}
I20260810 18:52:19.950968 20561 raft_consensus.cc:2726] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260810 18:52:19.983851 16701 leader_election.cc:334] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 3.309s (SENT)
I20260810 18:52:19.953564 20437 raft_consensus.cc:729] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:19.990820 16701 leader_election.cc:302] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:19.991858 20536 consensus_queue.cc:1035] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260810 18:52:19.992293 20439 raft_consensus.cc:504] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:19.993131 20592 raft_consensus.cc:2726] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:19.994364 20439 leader_election.cc:288] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:19.997169 20500 log.cc:924] Time spent T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 1.708s user 0.002s sys 0.002s
I20260810 18:52:19.979167 20458 raft_consensus.cc:482] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:19.999569 20458 raft_consensus.cc:504] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:20.001804 20458 leader_election.cc:288] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:20.003410 20437 consensus_queue.cc:260] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:20.004649 20437 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Advancing to term 6
W20260810 18:52:19.964784 17780 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55586 (request call id 352) took 1440 ms (client timeout 1905 ms). Trace:
W20260810 18:52:20.007437 17780 rpcz_store.cc:269] 0810 18:52:18.524764 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:18.525724 (+ 960us) service_pool.cc:227] Handling call
0810 18:52:19.953731 (+1428007us) spinlock_profiling.cc:243] Waited 397 ms on lock 0x7b54006a27d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:19.963424 (+ 9693us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":913843968}
W20260810 18:52:20.008001 20625 log.cc:924] Time spent T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.065s user 0.000s sys 0.001s
I20260810 18:52:19.953409 17771 raft_consensus.cc:3037] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:19.960009 20494 raft_consensus.cc:482] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:20.012028 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 105139us
I20260810 18:52:20.012538 20549 raft_consensus.cc:2726] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:20.012902 16447 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:20.015031 16695 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:20.023013 20594 raft_consensus.cc:465] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Not starting pre-election -- already a leader
I20260810 18:52:20.036336 17778 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55586 (request call id 344) took 4428 ms. Trace:
I20260810 18:52:20.037081 16632 raft_consensus.cc:1261] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260810 18:52:20.026610 17781 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55570 (request call id 450) took 4171 ms. Trace:
I20260810 18:52:20.026670 17780 raft_consensus.cc:1261] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260810 18:52:20.034808 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 71577.7us
I20260810 18:52:20.022190 16749 raft_consensus.cc:2454] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 7.
W20260810 18:52:20.035311 16575 connection.cc:232] RPC call timeout handler was delayed by -3.81336643290706s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1514s for a response
I20260810 18:52:20.012643 20494 raft_consensus.cc:504] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:20.042060 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:52:20.041896 16575 connection.cc:232] RPC call timeout handler was delayed by -3.7953664329834282s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1534s for a response
I20260810 18:52:20.041971 17727 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
W20260810 18:52:20.044500 16575 connection.cc:232] RPC call timeout handler was delayed by -3.7818664330989122s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1549s for a response
W20260810 18:52:20.044984 16575 connection.cc:232] RPC call timeout handler was delayed by -3.743166432948783s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1592s for a response
W20260810 18:52:20.045621 16575 connection.cc:232] RPC call timeout handler was delayed by -3.7089664330706s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.163s for a response
W20260810 18:52:20.046296 16575 connection.cc:232] RPC call timeout handler was delayed by -3.6792664329987019s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1663s for a response
W20260810 18:52:20.046806 16575 connection.cc:232] RPC call timeout handler was delayed by -3.6756664330605417s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16670000000000001s for a response
W20260810 18:52:20.047328 16575 connection.cc:232] RPC call timeout handler was delayed by -3.6144664329476655s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17350000000000002s for a response
W20260810 18:52:20.049188 16575 connection.cc:232] RPC call timeout handler was delayed by -3.5757664330303669s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1778s for a response
I20260810 18:52:20.040243 20591 consensus_queue.cc:1035] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260810 18:52:20.049885 16575 connection.cc:232] RPC call timeout handler was delayed by -3.5739664330612868s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17800000000000002s for a response
I20260810 18:52:20.051970 20110 raft_consensus.cc:2781] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader election won for term 7
I20260810 18:52:20.050952 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:20.053314 16626 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Advancing to term 7
W20260810 18:52:20.051457 16575 connection.cc:232] RPC call timeout handler was delayed by -3.5550664330367s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1801s for a response
W20260810 18:52:20.054193 16575 connection.cc:232] RPC call timeout handler was delayed by -3.55326643306762s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18030000000000002s for a response
I20260810 18:52:20.022569 20513 raft_consensus.cc:3032] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 LEADER]: Stepping down as leader of term 4
I20260810 18:52:20.052536 20110 raft_consensus.cc:686] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 7 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:20.044196 20556 raft_consensus.cc:3037] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
W20260810 18:52:20.054582 16575 connection.cc:232] RPC call timeout handler was delayed by -3.5478664329275489s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1809s for a response
I20260810 18:52:20.057521 16447 leader_election.cc:302] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:20.059690 16506 raft_consensus.cc:1226] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:20.057652 16450 leader_election.cc:302] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:20.033702 19858 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:20.067513 16575 connection.cc:232] RPC call timeout handler was delayed by -3.5388664330821484s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1819s for a response
W20260810 18:52:20.068907 16575 connection.cc:232] RPC call timeout handler was delayed by -3.5244664330966771s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1835s for a response
W20260810 18:52:20.069506 16575 connection.cc:232] RPC call timeout handler was delayed by -3.5019664329010993s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18600000000000003s for a response
W20260810 18:52:20.069949 16575 connection.cc:232] RPC call timeout handler was delayed by -3.3849664330482483s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.199s for a response
W20260810 18:52:20.070464 16575 connection.cc:232] RPC call timeout handler was delayed by -3.3759664329700172s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2s for a response
W20260810 18:52:20.070940 16575 connection.cc:232] RPC call timeout handler was delayed by -3.3507664329372346s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2028s for a response
W20260810 18:52:20.071650 16575 connection.cc:232] RPC call timeout handler was delayed by -3.3426664329599589s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2037s for a response
W20260810 18:52:20.072211 16575 connection.cc:232] RPC call timeout handler was delayed by -3.2922664331272244s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2093s for a response
W20260810 18:52:20.072723 16575 connection.cc:232] RPC call timeout handler was delayed by -3.2778664329089224s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2109s for a response
W20260810 18:52:20.073199 16575 connection.cc:232] RPC call timeout handler was delayed by -3.2517664330080152s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2138s for a response
W20260810 18:52:20.073691 16575 connection.cc:232] RPC call timeout handler was delayed by -3.2139664329588413s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21800000000000003s for a response
W20260810 18:52:20.074163 16575 connection.cc:232] RPC call timeout handler was delayed by -3.2058664329815656s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2189s for a response
W20260810 18:52:20.074644 16575 connection.cc:232] RPC call timeout handler was delayed by -3.1851664329878986s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22120000000000004s for a response
W20260810 18:52:20.075134 16575 connection.cc:232] RPC call timeout handler was delayed by -3.0051664330530912s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2412s for a response
W20260810 18:52:20.075618 16575 connection.cc:232] RPC call timeout handler was delayed by -2.9241664330475032s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2502s for a response
W20260810 18:52:20.076076 16575 connection.cc:232] RPC call timeout handler was delayed by -2.8188664331100881s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2619s for a response
W20260810 18:52:20.076624 16575 connection.cc:232] RPC call timeout handler was delayed by -2.604666433064267s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2857s for a response
W20260810 18:52:20.077078 16575 connection.cc:232] RPC call timeout handler was delayed by -2.1492664329707623s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33630000000000004s for a response
I20260810 18:52:20.089784 16575 leader_election.cc:302] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:20.060187 20513 raft_consensus.cc:729] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
W20260810 18:52:20.102288 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 44730.6us
I20260810 18:52:20.107303 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:20.097182 17726 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.925s (SENT) [suppressed 40 similar messages]
W20260810 18:52:20.124280 17726 leader_election.cc:334] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.058s (SENT)
I20260810 18:52:20.124953 17726 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:20.127188 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:20.128080 16495 raft_consensus.cc:2362] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5. Re-sending same reply.
W20260810 18:52:20.112813 20622 log.cc:924] Time spent T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.072s user 0.001s sys 0.000s
I20260810 18:52:20.092967 19940 raft_consensus.cc:2726] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:20.079653 19944 consensus_queue.cc:1035] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260810 18:52:20.066862 20124 consensus_queue.cc:1035] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260810 18:52:20.098676 19587 raft_consensus.cc:2726] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:20.138813 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:20.139541 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
W20260810 18:52:20.140483 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 58082.9us
I20260810 18:52:20.143167 16748 raft_consensus.cc:3037] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:20.139919 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:20.145789 16500 raft_consensus.cc:1226] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:20.148265 19945 consensus_queue.cc:1046] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260810 18:52:20.151628 20628 log.cc:924] Time spent T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.056s user 0.000s sys 0.001s
I20260810 18:52:20.137058 19996 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:20.113662 20184 raft_consensus.cc:2726] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:20.153512 19996 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:20.123344 20594 raft_consensus.cc:2781] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:52:20.155565 19996 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:20.155315 20110 consensus_queue.cc:237] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:20.156225 16495 raft_consensus.cc:1261] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260810 18:52:20.137365 20076 raft_consensus.cc:2726] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:20.037721 17781 rpcz_store.cc:276] 0810 18:52:15.855264 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:17.178280 (+1323016us) service_pool.cc:227] Handling call
0810 18:52:17.228932 (+ 50652us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:17.328662 (+ 99730us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:17.330112 (+ 1450us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:17.341658 (+ 11546us) log.cc:842] Serialized 26 byte log entry
0810 18:52:17.590326 (+248668us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:17.591876 (+ 1550us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:20.026179 (+2434303us) spinlock_profiling.cc:243] Waited 94.4 ms on lock 0x7b540069da54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:20.026250 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:20.026304 (+ 54us) raft_consensus.cc:1671] finished
0810 18:52:20.026316 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:20.026547 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2521,"raft.queue_time_us":3967,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":7564,"spinlock_wait_cycles":217133824,"thread_start_us":235619,"threads_started":1,"wal-append.queue_time_us":441523,"wal-append.run_cpu_time_us":2777,"wal-append.run_wall_time_us":1075220}
I20260810 18:52:20.131467 20037 consensus_queue.cc:1046] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260810 18:52:20.163815 16506 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1482) took 1595 ms. Trace:
I20260810 18:52:20.164389 16506 rpcz_store.cc:276] 0810 18:52:18.568443 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:18.568608 (+ 165us) service_pool.cc:227] Handling call
0810 18:52:18.602740 (+ 34132us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:20.067265 (+1464525us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:20.163524 (+ 96259us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b54001379d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:20.163741 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3350041344}
I20260810 18:52:20.163887 20037 raft_consensus.cc:3032] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Stepping down as leader of term 6
W20260810 18:52:20.163919 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 74489.6us
I20260810 18:52:20.174903 16695 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:20.177264 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:20.197569 20556 raft_consensus.cc:2726] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:20.190522 20444 raft_consensus.cc:2781] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:20.198490 16747 raft_consensus.cc:1226] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:20.198621 17724 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:20.199707 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:20.201418 20594 raft_consensus.cc:686] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:20.209151 19912 raft_consensus.cc:3032] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Stepping down as leader of term 4
I20260810 18:52:20.209537 19912 raft_consensus.cc:729] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:20.211236 17771 raft_consensus.cc:2454] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:20.198503 20444 raft_consensus.cc:686] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:20.213483 20444 consensus_queue.cc:237] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:20.036798 17778 rpcz_store.cc:276] 0810 18:52:15.607632 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:15.610284 (+ 2652us) service_pool.cc:227] Handling call
0810 18:52:15.708941 (+ 98657us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:15.726573 (+ 17632us) raft_consensus.cc:1520] Early marking committed up to index 6
0810 18:52:16.225439 (+498866us) log.cc:842] Serialized 14 byte log entry
0810 18:52:18.709305 (+2483866us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:18.729282 (+ 19977us) raft_consensus.cc:1631] Marking committed up to 6
0810 18:52:18.744421 (+ 15139us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:20.003087 (+1258666us) spinlock_profiling.cc:243] Waited 9.26 ms on lock 0x7b54006b03d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:20.015452 (+ 12365us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:20.036256 (+ 20804us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5052,"prepare.queue_time_us":1650694,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":41,"raft.queue_time_us":37901,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":21068,"spinlock_wait_cycles":21302656,"thread_start_us":2857220,"threads_started":2,"wal-append.queue_time_us":1549382,"wal-append.run_cpu_time_us":1437,"wal-append.run_wall_time_us":1021420}
I20260810 18:52:20.191604 20513 consensus_queue.cc:260] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:20.179085 20037 raft_consensus.cc:729] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:20.272480 16748 raft_consensus.cc:2454] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:20.196364 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:20.264305 19940 raft_consensus.cc:482] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:20.205050 20594 consensus_queue.cc:237] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:20.201612 16575 leader_election.cc:398] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 election: Vote denied by peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171) with higher term. Message: Invalid argument: T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 6. Current term is 7.
W20260810 18:52:20.268141 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 75831us
I20260810 18:52:20.276016 16695 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:20.268486 17724 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:20.278640 17726 leader_election.cc:334] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.161s (SENT)
I20260810 18:52:20.210335 19912 consensus_queue.cc:260] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:20.291563 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 64653.9us
I20260810 18:52:20.271735 16626 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
I20260810 18:52:20.162856 16345 catalog_manager.cc:5495] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:52:20.298969 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
W20260810 18:52:20.327462 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 334849us
W20260810 18:52:20.328456 16701 leader_election.cc:334] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.938s (SENT)
W20260810 18:52:20.341769 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 46480.8us
I20260810 18:52:20.271911 20076 raft_consensus.cc:2726] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260810 18:52:20.328948 16701 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:20.348361 20464 consensus_queue.cc:1035] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260810 18:52:20.333484 16345 catalog_manager.cc:5495] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:52:20.356082 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:20.292356 20459 raft_consensus.cc:2726] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260810 18:52:20.298390 20571 raft_consensus.cc:2781] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:20.366393 16750 raft_consensus.cc:2396] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260810 18:52:20.335295 20494 leader_election.cc:288] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:20.381127 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 179778us
I20260810 18:52:20.333832 20437 consensus_queue.cc:1035] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.040s
I20260810 18:52:20.293887 19940 raft_consensus.cc:504] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:20.386276 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 107686us
I20260810 18:52:20.338618 20373 consensus_queue.cc:1046] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20260810 18:52:20.378136 16345 catalog_manager.cc:5495] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 1 to 5, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 5 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:52:20.379592 19912 raft_consensus.cc:3037] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:20.333531 20591 consensus_queue.cc:1046] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.083s
I20260810 18:52:20.375739 20513 raft_consensus.cc:3037] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:20.397969 17778 raft_consensus.cc:1226] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260810 18:52:20.375788 20592 raft_consensus.cc:2726] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260810 18:52:20.392494 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 64034.1us
I20260810 18:52:20.379987 20571 raft_consensus.cc:686] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
W20260810 18:52:20.403546 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 48090.5us
W20260810 18:52:20.403777 16701 leader_election.cc:334] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.010s (SENT)
I20260810 18:52:20.388190 20459 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:20.358742 20458 consensus_queue.cc:1046] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.036s
I20260810 18:52:20.412847 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:20.410593 16345 catalog_manager.cc:5495] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260810 18:52:20.413506 16501 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:20.383494 20556 consensus_queue.cc:1046] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.028s
I20260810 18:52:20.425359 16345 catalog_manager.cc:5495] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 1 to 3, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 3 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } }
I20260810 18:52:20.396997 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:20.427290 17771 raft_consensus.cc:2396] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260810 18:52:20.427335 16345 catalog_manager.cc:5495] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 2 to 5, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 5 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } }
I20260810 18:52:20.429291 16345 catalog_manager.cc:5495] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } }
I20260810 18:52:20.429677 20606 consensus_queue.cc:1046] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260810 18:52:20.430743 20606 raft_consensus.cc:3032] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Stepping down as leader of term 4
I20260810 18:52:20.431103 20606 raft_consensus.cc:729] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:20.431285 16345 catalog_manager.cc:5495] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 1 to 3, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 3 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } }
W20260810 18:52:20.432180 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 56576.2us
I20260810 18:52:20.382185 20415 consensus_queue.cc:1035] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.020s
I20260810 18:52:20.436908 16501 raft_consensus.cc:2454] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 7.
I20260810 18:52:20.439881 16345 catalog_manager.cc:5495] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:52:20.419258 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:20.441632 16495 raft_consensus.cc:2454] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 5.
I20260810 18:52:20.420799 16701 leader_election.cc:302] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:20.422546 16508 raft_consensus.cc:1226] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:20.432928 16450 leader_election.cc:302] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:20.445710 16345 catalog_manager.cc:5495] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 1 to 3, leader changed from d304ac47fdf846de81337057a4ef7bea (127.14.32.193) to 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196). New cstate: current_term: 3 leader_uuid: "831925dd8b644ebab7eb0d672a94eddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:52:20.446450 16447 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:20.432682 20464 raft_consensus.cc:3032] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:52:20.451289 16575 leader_election.cc:302] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:20.434064 20536 raft_consensus.cc:3032] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:52:20.440703 20415 consensus_queue.cc:1225] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer d304ac47fdf846de81337057a4ef7bea log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260810 18:52:20.444175 20556 consensus_queue.cc:1225] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer d304ac47fdf846de81337057a4ef7bea log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260810 18:52:20.402545 20037 consensus_queue.cc:260] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:20.413476 20459 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:20.464301 16345 catalog_manager.cc:5495] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 1 to 3, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 3 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } }
I20260810 18:52:20.452912 20561 raft_consensus.cc:2726] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:20.484357 16345 catalog_manager.cc:5495] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 3 to 5, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196). New cstate: current_term: 5 leader_uuid: "831925dd8b644ebab7eb0d672a94eddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:52:20.460268 20536 raft_consensus.cc:729] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:20.487298 20316 raft_consensus.cc:2781] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:52:20.487545 20037 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:20.468690 20458 raft_consensus.cc:2726] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260810 18:52:20.489384 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 48381.2us
I20260810 18:52:20.490326 16575 leader_election.cc:398] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Vote denied by peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171) with higher term. Message: Invalid argument: T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6.
I20260810 18:52:20.493726 16450 leader_election.cc:302] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:20.495399 20316 raft_consensus.cc:686] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:20.496236 19940 leader_election.cc:288] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:20.496263 20316 consensus_queue.cc:237] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:20.499998 20549 consensus_queue.cc:1035] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:52:20.500288 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:20.501298 17781 raft_consensus.cc:2396] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260810 18:52:20.474654 19996 raft_consensus.cc:2726] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:20.454052 20464 raft_consensus.cc:729] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
W20260810 18:52:20.520131 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 116380us
I20260810 18:52:20.493475 16345 catalog_manager.cc:5495] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 1 to 5, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196). New cstate: current_term: 5 leader_uuid: "831925dd8b644ebab7eb0d672a94eddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
W20260810 18:52:20.521450 16701 leader_election.cc:334] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.201s (SENT)
I20260810 18:52:20.494863 19935 raft_consensus.cc:2726] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:20.510210 16448 leader_election.cc:302] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:20.532649 16701 leader_election.cc:302] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:20.526901 17724 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:20.530177 20464 consensus_queue.cc:260] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:20.529771 16345 catalog_manager.cc:5495] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 1 to 3, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 3 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } }
I20260810 18:52:20.531241 16500 raft_consensus.cc:1226] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:20.449288 20571 consensus_queue.cc:237] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:20.532238 16504 pending_rounds.cc:76] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea: Aborting all ops after (but not including) 5
I20260810 18:52:20.541806 16504 pending_rounds.cc:97] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260810 18:52:20.542115 16504 raft_consensus.cc:2866] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:52:20.536448 20437 raft_consensus.cc:2781] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:20.543249 20437 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:20.543645 20437 raft_consensus.cc:3037] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:20.543687 20459 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:20.497587 20536 consensus_queue.cc:260] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:20.536859 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:20.444983 20606 consensus_queue.cc:260] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:20.556433 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 46327.8us
I20260810 18:52:20.563800 16345 catalog_manager.cc:5495] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 1 to 3, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 3 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } }
W20260810 18:52:20.567131 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 75815.1us
I20260810 18:52:20.566824 16345 catalog_manager.cc:5495] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be reported cstate change: term changed from 1 to 3, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 3 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } }
I20260810 18:52:20.569736 20561 raft_consensus.cc:2726] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:20.579811 16506 raft_consensus.cc:2396] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260810 18:52:20.553465 20595 consensus_queue.cc:1046] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20260810 18:52:20.581483 16500 raft_consensus.cc:1261] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260810 18:52:20.593102 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:20.555629 20592 raft_consensus.cc:3032] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 LEADER]: Stepping down as leader of term 4
I20260810 18:52:20.558163 20373 raft_consensus.cc:3032] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:20.560642 20549 raft_consensus.cc:2781] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election won for term 7
W20260810 18:52:20.605537 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 84095.7us
W20260810 18:52:20.609380 16701 leader_election.cc:334] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.211s (SENT)
I20260810 18:52:20.609869 16701 leader_election.cc:302] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:20.606580 16575 leader_election.cc:302] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:20.596637 19863 consensus_queue.cc:1035] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 3, Time since last communication: 0.017s
I20260810 18:52:20.616003 20464 raft_consensus.cc:3037] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:20.603623 20536 raft_consensus.cc:3037] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:20.597005 20592 raft_consensus.cc:729] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:20.602819 20373 raft_consensus.cc:729] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:20.623426 20437 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:20.641410 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 43577.7us
W20260810 18:52:20.642513 16701 leader_election.cc:334] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 3.729s (SENT)
I20260810 18:52:20.616055 20494 raft_consensus.cc:2726] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:20.612468 20540 raft_consensus.cc:2726] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:20.652746 16701 leader_election.cc:302] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:20.668768 16742 raft_consensus.cc:1261] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260810 18:52:20.670048 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 47304.4us
I20260810 18:52:20.673604 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:20.647013 20549 raft_consensus.cc:686] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:20.681882 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:20.685993 19863 consensus_queue.cc:1035] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 18:52:20.678345 16575 leader_election.cc:302] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:20.701460 20654 log.cc:924] Time spent T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260810 18:52:20.697203 20458 raft_consensus.cc:2726] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260810 18:52:20.702472 20651 log.cc:924] Time spent T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
W20260810 18:52:20.703081 20653 log.cc:924] Time spent T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.183s user 0.000s sys 0.001s
I20260810 18:52:20.701304 16575 leader_election.cc:302] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:20.693193 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:20.717248 16575 leader_election.cc:302] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:20.719647 16495 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "681439c754d942b7847d73744abbe269"
dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
mode: GRACEFUL
new_leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
from {username='slave'} at 127.0.0.1:51572
I20260810 18:52:20.720491 16495 raft_consensus.cc:595] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Received request to transfer leadership to TS 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:20.672279 20556 raft_consensus.cc:2726] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:20.732259 17787 raft_consensus.cc:1226] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:20.737790 20458 raft_consensus.cc:2726] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:20.782559 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:20.737610 20439 raft_consensus.cc:2726] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:20.752106 20459 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:20.688772 20373 consensus_queue.cc:260] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:20.808503 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 89440.5us
I20260810 18:52:20.731930 20592 consensus_queue.cc:260] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:20.996413 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 240793us
W20260810 18:52:21.075510 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 360792us
I20260810 18:52:20.809239 20549 consensus_queue.cc:237] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:21.205412 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 418571us
I20260810 18:52:21.184042 20373 raft_consensus.cc:3037] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:21.200956 20592 raft_consensus.cc:3037] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
W20260810 18:52:21.186771 20464 consensus_meta.cc:287] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.565s user 0.011s sys 0.002s
W20260810 18:52:21.188498 20658 log.cc:924] Time spent T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.408s user 0.005s sys 0.005s
I20260810 18:52:21.188264 20459 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:20.788403 20494 consensus_queue.cc:1046] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.032s
I20260810 18:52:21.190547 16622 raft_consensus.cc:1261] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260810 18:52:21.199174 20536 consensus_meta.cc:287] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.555s user 0.002s sys 0.014s
W20260810 18:52:21.030264 20096 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.018s sys 0.041s
I20260810 18:52:20.747633 20561 consensus_queue.cc:1035] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.055s
I20260810 18:52:21.230199 16575 leader_election.cc:302] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:21.231380 16701 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:21.233551 20037 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:21.251268 20037 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:21.253741 20037 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:21.235172 16746 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1"
dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
mode: GRACEFUL
new_leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
from {username='slave'} at 127.0.0.1:45870
I20260810 18:52:21.254585 16746 raft_consensus.cc:595] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Received request to transfer leadership to TS 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:21.255012 16746 raft_consensus.cc:603] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260810 18:52:21.211004 20643 log.cc:924] Time spent T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.427s user 0.002s sys 0.010s
I20260810 18:52:21.194088 20437 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:21.216142 20444 raft_consensus.cc:482] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:21.250200 20096 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.748s user 0.018s sys 0.045s
I20260810 18:52:21.262526 20037 raft_consensus.cc:2726] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:21.262336 20571 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:21.269860 16746 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb"
dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
mode: GRACEFUL
new_leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
from {username='slave'} at 127.0.0.1:45870
W20260810 18:52:21.270255 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 59077.4us
I20260810 18:52:21.270089 20571 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:21.268801 20444 raft_consensus.cc:504] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:21.271158 16575 leader_election.cc:398] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 election: Vote denied by peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171) with higher term. Message: Invalid argument: T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6.
I20260810 18:52:21.270524 16746 raft_consensus.cc:595] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Received request to transfer leadership to TS 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:21.272403 20571 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:21.272614 16575 leader_election.cc:302] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:21.268647 20459 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:21.273591 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:21.274575 20494 raft_consensus.cc:2726] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:21.274111 16746 raft_consensus.cc:603] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260810 18:52:21.278620 20655 log.cc:924] Time spent T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260810 18:52:21.280128 16701 leader_election.cc:302] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:21.280836 16742 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9e960488563344af83b979c260c2dd06"
dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
mode: GRACEFUL
new_leader_uuid: "831925dd8b644ebab7eb0d672a94eddd"
from {username='slave'} at 127.0.0.1:45870
I20260810 18:52:21.281181 20037 raft_consensus.cc:2726] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:21.281486 16742 raft_consensus.cc:595] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Received request to transfer leadership to TS 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:21.281828 16742 raft_consensus.cc:603] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260810 18:52:21.284077 16701 leader_election.cc:302] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:21.291669 20037 raft_consensus.cc:2781] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:21.292339 20037 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:21.263052 20464 raft_consensus.cc:2726] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:52:21.293324 20037 raft_consensus.cc:3037] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:21.277484 16575 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:21.284861 16742 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "da1fa324d17c479383225a8cea494a80"
dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
mode: GRACEFUL
new_leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
from {username='slave'} at 127.0.0.1:45870
I20260810 18:52:21.300300 16742 raft_consensus.cc:595] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Received request to transfer leadership to TS 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:21.292160 16701 leader_election.cc:302] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:21.274487 17781 raft_consensus.cc:2454] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
W20260810 18:52:21.299593 16631 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.579s user 0.001s sys 0.016s
I20260810 18:52:21.300740 16742 raft_consensus.cc:603] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260810 18:52:21.276083 20444 leader_election.cc:288] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:21.310580 16631 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.590s user 0.002s sys 0.016s
I20260810 18:52:21.317507 16701 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:21.327080 16695 leader_election.cc:302] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:21.317237 20571 raft_consensus.cc:2726] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:21.328263 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260810 18:52:21.342675 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 65390.3us
I20260810 18:52:21.323225 20689 raft_consensus.cc:2781] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:21.340726 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:52:21.345196 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:21.330807 20444 raft_consensus.cc:2781] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260810 18:52:21.341801 20513 consensus_meta.cc:287] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.936s user 0.014s sys 0.007s
I20260810 18:52:21.339028 20683 raft_consensus.cc:2781] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260810 18:52:21.369563 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 67603.6us
I20260810 18:52:21.374116 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:52:21.348591 20689 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:21.386543 16701 leader_election.cc:302] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:21.359048 20444 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:21.367266 20683 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:21.399168 20037 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:21.400529 20660 log.cc:924] Time spent T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260810 18:52:21.400755 16575 leader_election.cc:302] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:21.392535 20689 raft_consensus.cc:3037] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:21.395642 20513 raft_consensus.cc:2726] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:21.397346 20606 raft_consensus.cc:3037] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:21.411841 16701 leader_election.cc:302] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:21.402184 20444 raft_consensus.cc:3037] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:21.423691 16701 leader_election.cc:398] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Vote denied by peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827) with higher term. Message: Invalid argument: T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3.
I20260810 18:52:21.424178 16701 leader_election.cc:401] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20260810 18:52:21.437386 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 100384us
I20260810 18:52:21.443672 16701 leader_election.cc:302] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:21.410019 20683 raft_consensus.cc:3037] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
W20260810 18:52:21.465251 20698 log.cc:924] Time spent T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.058s user 0.000s sys 0.001s
I20260810 18:52:21.447899 20540 raft_consensus.cc:2726] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:21.479838 16750 raft_consensus.cc:1261] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260810 18:52:21.481979 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 81342.7us
I20260810 18:52:21.483305 16575 leader_election.cc:302] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:21.484370 20458 raft_consensus.cc:2741] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20260810 18:52:21.452059 20037 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:21.453298 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:21.418263 20571 raft_consensus.cc:2726] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:52:21.440074 20513 raft_consensus.cc:2726] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827) with higher term. Message: Invalid argument: T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 2. Current term is 3.
I20260810 18:52:21.439563 20464 raft_consensus.cc:2726] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260810 18:52:21.493000 16701 connection.cc:232] RPC call timeout handler was delayed by -1.3897255558986217s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
W20260810 18:52:21.498872 16701 connection.cc:232] RPC call timeout handler was delayed by -1.0186093559022993s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17400000000000002s for a response
I20260810 18:52:21.498620 20689 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:21.508875 20464 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:21.509440 20464 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:21.510344 20689 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:21.511547 20464 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:21.514807 16575 leader_election.cc:302] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:21.532478 20689 raft_consensus.cc:482] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:21.533144 20689 raft_consensus.cc:504] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:21.533625 20444 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:21.564407 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 115222us
W20260810 18:52:21.565145 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 101658us
I20260810 18:52:21.531963 20464 raft_consensus.cc:2726] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260810 18:52:21.572937 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 58281us
W20260810 18:52:21.575037 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 183129us
I20260810 18:52:21.580753 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:21.581403 16501 raft_consensus.cc:2454] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 5.
I20260810 18:52:21.586575 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:21.591542 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:21.591789 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:21.592321 16506 raft_consensus.cc:3037] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:21.593530 16742 raft_consensus.cc:1226] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:21.592185 16508 raft_consensus.cc:2454] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
I20260810 18:52:21.597451 20110 consensus_queue.cc:1046] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 18:52:21.598341 20110 raft_consensus.cc:3032] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:21.598675 20110 raft_consensus.cc:729] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:21.599689 20110 consensus_queue.cc:260] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:21.600644 20110 raft_consensus.cc:3037] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:21.602445 16500 raft_consensus.cc:1261] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260810 18:52:21.602895 16500 pending_rounds.cc:76] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea: Aborting all ops after (but not including) 5
I20260810 18:52:21.601011 20278 consensus_queue.cc:1035] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:52:21.603120 16500 pending_rounds.cc:97] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260810 18:52:21.603415 16500 raft_consensus.cc:2866] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:52:21.601244 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:21.602260 16508 raft_consensus.cc:1261] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260810 18:52:21.604380 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:21.604540 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:21.598502 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:21.605145 16507 raft_consensus.cc:2454] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:21.604432 16505 raft_consensus.cc:2379] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 6: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:21.573734 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:21.607407 16573 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:21.615332 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 45736.2us
W20260810 18:52:21.614678 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 120283us
I20260810 18:52:21.623203 16575 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:21.624456 20458 raft_consensus.cc:2726] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:21.625933 16747 raft_consensus.cc:1261] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260810 18:52:21.627454 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:21.621939 20373 raft_consensus.cc:2781] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:21.629143 20373 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:21.630447 17724 leader_election.cc:302] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:21.629586 20373 raft_consensus.cc:3037] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:21.636687 20693 raft_consensus.cc:2781] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:21.621744 20513 consensus_queue.cc:1035] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260810 18:52:21.637120 20693 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:21.637507 20693 raft_consensus.cc:3037] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:21.640276 16575 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:21.644227 20373 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:21.645533 20458 raft_consensus.cc:2781] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:21.645943 20458 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:21.646346 20458 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:21.646536 20373 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:21.651206 20458 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:21.653090 20693 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:21.654263 16575 leader_election.cc:302] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:21.657119 20458 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:21.631228 20592 consensus_queue.cc:1035] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
I20260810 18:52:21.657689 20373 raft_consensus.cc:2726] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:21.658370 20693 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:21.657240 16575 leader_election.cc:302] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:21.659343 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:21.660161 16575 leader_election.cc:302] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:21.660867 20439 raft_consensus.cc:2726] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:52:21.661484 20439 raft_consensus.cc:2726] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:52:21.662801 16575 leader_election.cc:302] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:21.663637 20439 raft_consensus.cc:2726] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:21.672001 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:21.672637 16507 raft_consensus.cc:3037] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:21.615692 20571 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:21.675642 20571 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:21.677719 20571 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:21.677845 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:21.678495 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:21.681169 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:21.687279 16573 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:21.688151 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:21.688705 16501 raft_consensus.cc:3037] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:21.690377 20494 raft_consensus.cc:2726] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260810 18:52:21.694131 16506 raft_consensus.cc:2454] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
I20260810 18:52:21.633066 17781 raft_consensus.cc:2454] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
I20260810 18:52:21.696895 16501 raft_consensus.cc:2454] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 7.
I20260810 18:52:21.698241 16573 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:21.699271 20494 raft_consensus.cc:2781] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader election won for term 7
I20260810 18:52:21.699781 20494 raft_consensus.cc:686] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 7 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:21.696275 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:21.700646 20494 consensus_queue.cc:237] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:21.701303 17787 raft_consensus.cc:2454] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:21.702407 16695 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
W20260810 18:52:21.703615 17726 leader_election.cc:334] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.726s (SENT)
W20260810 18:52:21.703747 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 67318.3us
I20260810 18:52:21.704064 17726 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:21.704171 20513 raft_consensus.cc:2781] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:21.704635 20513 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:21.704895 20437 raft_consensus.cc:2726] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:21.705001 20513 raft_consensus.cc:3037] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:21.705997 17724 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:21.706835 20437 raft_consensus.cc:2781] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:21.707768 20437 raft_consensus.cc:686] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 6 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:21.708675 20437 consensus_queue.cc:237] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:21.591172 20444 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:21.717918 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:21.718506 16506 raft_consensus.cc:2454] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:21.729503 16507 raft_consensus.cc:2454] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
I20260810 18:52:21.730760 17724 leader_election.cc:302] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:21.717006 20689 leader_election.cc:288] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:21.773049 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 110389us
I20260810 18:52:21.727254 20464 raft_consensus.cc:482] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:21.773814 19587 consensus_queue.cc:1035] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.003s
I20260810 18:52:21.776651 16575 leader_election.cc:302] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:21.749781 20571 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:21.756004 20415 raft_consensus.cc:2781] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader election won for term 6
W20260810 18:52:21.786021 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 166759us
I20260810 18:52:21.772126 20584 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:21.790302 16701 leader_election.cc:334] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.161s (SENT)
I20260810 18:52:21.791224 16701 leader_election.cc:302] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:21.793915 20592 raft_consensus.cc:2726] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:21.799947 20458 raft_consensus.cc:482] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:21.800642 20591 raft_consensus.cc:3032] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:21.800570 20458 raft_consensus.cc:504] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:21.802847 20458 leader_election.cc:288] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:21.803462 20591 raft_consensus.cc:729] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:21.805409 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:21.805035 20591 consensus_queue.cc:260] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:21.806450 16502 raft_consensus.cc:1226] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:21.814639 16631 raft_consensus.cc:1226] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:21.812415 20362 raft_consensus.cc:979] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea: : Instructing follower 0397d760589d40d3b92ef141cdfc2dbb to start an election
I20260810 18:52:21.815683 20362 raft_consensus.cc:1067] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Signalling peer 0397d760589d40d3b92ef141cdfc2dbb to start an election
I20260810 18:52:21.816702 20458 consensus_queue.cc:1046] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 18:52:21.818889 20591 raft_consensus.cc:3037] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:21.820348 16621 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "681439c754d942b7847d73744abbe269"
dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
from {username='slave'} at 127.0.0.1:53870
I20260810 18:52:21.820922 16621 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:52:21.821260 16621 raft_consensus.cc:3037] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:21.822353 17783 raft_consensus.cc:1261] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260810 18:52:21.824719 20444 consensus_queue.cc:1035] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 18:52:21.830756 20164 consensus_queue.cc:1046] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:52:21.832222 20164 raft_consensus.cc:3032] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:21.783836 20689 raft_consensus.cc:2726] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:21.841995 16575 leader_election.cc:302] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:21.843003 20319 consensus_queue.cc:1035] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.056s
I20260810 18:52:21.832700 20164 raft_consensus.cc:729] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:21.845456 20164 consensus_queue.cc:260] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:21.840613 16621 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:21.855469 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:21.856087 16502 raft_consensus.cc:3032] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:21.796599 20415 raft_consensus.cc:686] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 6 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:21.834061 20584 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:21.861475 20415 consensus_queue.cc:237] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:21.856354 16502 raft_consensus.cc:729] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:21.862560 20584 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:21.863302 16502 consensus_queue.cc:260] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:21.864372 16502 raft_consensus.cc:3037] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:21.866999 16621 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:21.782109 20464 raft_consensus.cc:504] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:21.874747 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:21.891325 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:21.790746 20571 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:21.797782 20683 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:21.901917 16622 raft_consensus.cc:1261] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260810 18:52:21.885193 20458 raft_consensus.cc:2726] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:21.887224 17781 raft_consensus.cc:3037] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Advancing to term 6
W20260810 18:52:21.914230 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 78721.9us
I20260810 18:52:21.917106 20464 leader_election.cc:288] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:21.918908 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:21.920691 20549 consensus_queue.cc:1035] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:52:21.928514 20683 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:21.929404 16745 raft_consensus.cc:2379] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 6: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:21.936527 16575 leader_election.cc:302] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:21.914283 20164 raft_consensus.cc:3037] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:21.950170 20571 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:21.959468 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 44417us
I20260810 18:52:21.926800 20513 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:21.960428 16575 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:21.961620 20458 raft_consensus.cc:2726] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:21.962333 20513 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:21.962864 16575 leader_election.cc:302] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:21.964463 20464 raft_consensus.cc:2726] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:21.962337 20458 raft_consensus.cc:2726] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:21.968246 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:21.986305 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:21.987098 17780 raft_consensus.cc:2454] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:21.986351 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:21.989145 17787 raft_consensus.cc:3037] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:21.989341 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:21.990099 16505 raft_consensus.cc:2379] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 6: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:52:22.004091 16749 raft_consensus.cc:1226] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:21.968918 16575 leader_election.cc:302] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:22.012475 20458 raft_consensus.cc:2726] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:22.011945 16502 raft_consensus.cc:2454] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
W20260810 18:52:22.012825 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 44076us
I20260810 18:52:22.029656 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:22.030532 16505 raft_consensus.cc:2454] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:22.033047 16575 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:22.039120 20606 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:22.040809 16695 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:22.043785 16573 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:22.044935 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 45500.7us
I20260810 18:52:22.053131 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:22.053791 16507 raft_consensus.cc:3037] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 6
I20260810 18:52:22.058832 20561 raft_consensus.cc:2781] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:22.059263 20561 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:22.059600 20561 raft_consensus.cc:3037] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:22.064224 16507 raft_consensus.cc:2454] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:22.072479 16695 leader_election.cc:302] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:22.083190 16631 raft_consensus.cc:1261] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260810 18:52:22.084846 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 51922.6us
I20260810 18:52:22.085835 16575 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:22.104628 16575 leader_election.cc:302] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:22.105511 17783 raft_consensus.cc:1226] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 6 LEADER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:22.039695 20606 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:22.108141 16623 raft_consensus.cc:1226] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:22.113673 16575 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:22.114723 16575 leader_election.cc:302] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:22.089037 20592 raft_consensus.cc:2726] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260810 18:52:22.121146 16342 catalog_manager.cc:5495] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 3 to 5, now has a pending config: VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193), VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194), VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195), VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196). New cstate: current_term: 5 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: NON_VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: true } } } pending_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:52:22.127360 16695 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:22.127162 17787 raft_consensus.cc:2454] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:22.103626 20464 raft_consensus.cc:2726] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:22.107393 20439 raft_consensus.cc:2781] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:22.106638 20689 raft_consensus.cc:2726] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:22.109105 20494 raft_consensus.cc:2726] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:22.138901 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.109675 20095 consensus_queue.cc:1046] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20260810 18:52:22.140563 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.142659 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:22.125710 20536 raft_consensus.cc:2726] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260810 18:52:22.116577 17781 raft_consensus.cc:2454] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
I20260810 18:52:22.145424 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:22.146970 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.139004 16342 catalog_manager.cc:5495] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 1 to 4, leader changed from d304ac47fdf846de81337057a4ef7bea (127.14.32.193) to 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196). New cstate: current_term: 4 leader_uuid: "831925dd8b644ebab7eb0d672a94eddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:52:22.149019 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.149263 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:22.149829 16636 raft_consensus.cc:2396] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260810 18:52:22.150646 16622 raft_consensus.cc:1261] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260810 18:52:22.150734 16636 raft_consensus.cc:1226] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:22.136996 20373 raft_consensus.cc:2726] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:22.151441 16626 raft_consensus.cc:1261] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
I20260810 18:52:22.151808 16626 pending_rounds.cc:76] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb: Aborting all ops after (but not including) 4
I20260810 18:52:22.152123 16626 pending_rounds.cc:97] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20260810 18:52:22.152367 16626 raft_consensus.cc:2866] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:52:22.152915 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:22.154198 16345 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37030 (request call id 203) took 14965 ms (client timeout 15000 ms). Trace:
I20260810 18:52:22.154774 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.155454 16633 raft_consensus.cc:2454] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
I20260810 18:52:22.156333 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:22.156419 17726 leader_election.cc:334] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.925s (SENT)
I20260810 18:52:22.136629 20592 raft_consensus.cc:2781] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:22.157135 17726 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:22.177675 20592 raft_consensus.cc:686] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:22.177554 20184 consensus_queue.cc:1035] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:52:22.178730 20592 consensus_queue.cc:237] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:22.176771 16342 catalog_manager.cc:5495] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:52:22.181798 20693 raft_consensus.cc:2726] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260810 18:52:22.188803 16342 catalog_manager.cc:5495] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "831925dd8b644ebab7eb0d672a94eddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:52:22.190153 16342 catalog_manager.cc:5495] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 1 to 4, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196). New cstate: current_term: 4 leader_uuid: "831925dd8b644ebab7eb0d672a94eddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:52:22.191488 20439 raft_consensus.cc:686] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:22.191533 16342 catalog_manager.cc:5495] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 2 to 5, leader changed from 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 5 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:52:22.192605 20439 consensus_queue.cc:237] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:22.193661 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.194001 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.194653 16623 raft_consensus.cc:2454] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 3.
W20260810 18:52:22.154618 16345 rpcz_store.cc:269] 0810 18:52:07.189092 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:07.189435 (+ 343us) service_pool.cc:227] Handling call
0810 18:52:22.154097 (+14964662us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 18:52:21.459452 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 18:52:21.460230 (+ 778us) write_op.cc:432] Acquiring schema lock in shared mode
0810 18:52:21.460296 (+ 66us) write_op.cc:435] Acquired schema lock
0810 18:52:21.460338 (+ 42us) tablet.cc:660] Decoding operations
0810 18:52:21.462336 (+ 1998us) write_op.cc:620] Acquiring the partition lock for write op
0810 18:52:21.462485 (+ 149us) write_op.cc:641] Partition lock acquired for write op
0810 18:52:21.462530 (+ 45us) tablet.cc:683] Acquiring locks for 16 operations
0810 18:52:21.555176 (+ 92646us) tablet.cc:699] Row locks acquired
0810 18:52:21.556235 (+ 1059us) write_op.cc:260] PREPARE: finished
0810 18:52:21.556420 (+ 185us) write_op.cc:270] Start()
0810 18:52:21.556618 (+ 198us) write_op.cc:276] Timestamp: P: 1786387941556340 usec, L: 0
0810 18:52:21.556675 (+ 57us) op_driver.cc:331] REPLICATION: starting
0810 18:52:21.666901 (+110226us) log.cc:842] Serialized 6332 byte log entry
0810 18:52:21.799939 (+133038us) op_driver.cc:444] REPLICATION: finished
0810 18:52:21.823591 (+ 23652us) write_op.cc:301] APPLY: starting
0810 18:52:21.823739 (+ 148us) tablet.cc:1365] starting BulkCheckPresence
0810 18:52:21.824213 (+ 474us) tablet.cc:1368] finished BulkCheckPresence
0810 18:52:21.824422 (+ 209us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 18:52:21.915820 (+ 91398us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 18:52:21.916250 (+ 430us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=16
0810 18:52:21.916296 (+ 46us) write_op.cc:312] APPLY: finished
0810 18:52:21.916766 (+ 470us) log.cc:842] Serialized 113 byte log entry
0810 18:52:21.916949 (+ 183us) write_op.cc:489] Releasing partition, row and schema locks
0810 18:52:21.917119 (+ 170us) write_op.cc:454] Released schema lock
0810 18:52:21.917192 (+ 73us) write_op.cc:341] FINISH: Updating metrics
0810 18:52:21.917490 (+ 298us) write_op.cc:489] Releasing partition, row and schema locks
0810 18:52:21.917523 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":16736,"reported_tablets":38,"child_traces":[["op",{"apply.queue_time_us":22861,"apply.run_cpu_time_us":6228,"apply.run_wall_time_us":94795,"num_ops":16,"prepare.queue_time_us":202896,"prepare.run_cpu_time_us":23901,"prepare.run_wall_time_us":305097,"raft.queue_time_us":35008,"raft.run_cpu_time_us":4801,"raft.run_wall_time_us":23693,"replication_time_us":243144,"thread_start_us":277524,"threads_started":4,"wal-append.queue_time_us":94247}]]}
I20260810 18:52:22.196250 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.197297 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.198380 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:22.201144 20095 raft_consensus.cc:3032] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Stepping down as leader of term 4
I20260810 18:52:22.205122 20095 raft_consensus.cc:729] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:22.206089 20095 consensus_queue.cc:260] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:22.207420 20095 raft_consensus.cc:3037] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:22.212083 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.215813 16624 raft_consensus.cc:1226] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 4. Current term is 6. Ops: []
I20260810 18:52:22.223811 20561 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:22.226502 20606 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:22.226863 20247 consensus_queue.cc:1046] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260810 18:52:22.232807 17787 raft_consensus.cc:3037] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Advancing to term 7
W20260810 18:52:22.233652 20751 log.cc:924] Time spent T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260810 18:52:22.234328 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:22.235349 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.239293 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:22.240429 16743 raft_consensus.cc:1261] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260810 18:52:22.241781 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.243992 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.244280 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.245929 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.246989 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:22.257400 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.262694 16626 raft_consensus.cc:2454] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 3.
I20260810 18:52:22.259984 16623 raft_consensus.cc:2454] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
I20260810 18:52:22.264569 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.271044 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.271567 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.280459 20415 consensus_queue.cc:1035] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260810 18:52:22.281632 17726 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:22.270000 20713 raft_consensus.cc:3032] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:22.282436 20713 raft_consensus.cc:729] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:22.283327 20693 raft_consensus.cc:2726] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:22.286077 20561 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:22.287300 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:22.287518 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:22.288695 16749 raft_consensus.cc:2396] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260810 18:52:22.289697 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:22.296305 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:22.296905 17783 raft_consensus.cc:3037] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:22.297868 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:22.298866 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:22.300343 16450 leader_election.cc:302] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:22.299011 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:22.303043 19587 consensus_queue.cc:1035] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20260810 18:52:22.308422 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:22.303475 16505 raft_consensus.cc:1261] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260810 18:52:22.303695 20362 raft_consensus.cc:2726] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:22.304170 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:22.304551 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:22.311147 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:22.305359 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:22.307808 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
W20260810 18:52:22.318910 17726 leader_election.cc:334] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.927s (SENT)
I20260810 18:52:22.319406 16502 raft_consensus.cc:3037] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:22.320969 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.321260 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.323298 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.324589 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.319422 17726 leader_election.cc:302] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:22.328815 17726 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:22.329877 17787 raft_consensus.cc:1261] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260810 18:52:22.331714 16502 raft_consensus.cc:2454] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:22.334421 16575 leader_election.cc:302] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:22.335592 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:22.336072 16623 raft_consensus.cc:3037] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:22.338481 16695 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:22.341532 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:22.342406 16575 leader_election.cc:302] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:22.342844 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.343786 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.340133 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.340443 16626 pending_rounds.cc:76] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb: Aborting all ops after (but not including) 5
I20260810 18:52:22.346673 16626 pending_rounds.cc:97] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260810 18:52:22.347002 16626 raft_consensus.cc:2866] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:52:22.352447 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.377450 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.345793 20683 raft_consensus.cc:2781] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:22.377496 16502 raft_consensus.cc:1261] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260810 18:52:22.354344 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.342793 20373 raft_consensus.cc:2726] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260810 18:52:22.356982 16342 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55260 (request call id 123) took 14442 ms (client timeout 15000 ms). Trace:
I20260810 18:52:22.337543 20415 raft_consensus.cc:2726] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:22.344687 20591 consensus_queue.cc:1035] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.019s
I20260810 18:52:22.340106 20717 raft_consensus.cc:2726] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260810 18:52:22.383235 16342 rpcz_store.cc:269] 0810 18:52:07.914706 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:07.915998 (+ 1292us) service_pool.cc:227] Handling call
0810 18:52:22.356892 (+14440894us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 18:52:22.197693 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 18:52:22.198161 (+ 468us) write_op.cc:432] Acquiring schema lock in shared mode
0810 18:52:22.198200 (+ 39us) write_op.cc:435] Acquired schema lock
0810 18:52:22.198215 (+ 15us) tablet.cc:660] Decoding operations
0810 18:52:22.198971 (+ 756us) write_op.cc:620] Acquiring the partition lock for write op
0810 18:52:22.199021 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0810 18:52:22.199046 (+ 25us) tablet.cc:683] Acquiring locks for 6 operations
0810 18:52:22.209978 (+ 10932us) tablet.cc:699] Row locks acquired
0810 18:52:22.210005 (+ 27us) write_op.cc:260] PREPARE: finished
0810 18:52:22.210211 (+ 206us) write_op.cc:270] Start()
0810 18:52:22.210366 (+ 155us) write_op.cc:276] Timestamp: P: 1786387942210153 usec, L: 0
0810 18:52:22.210388 (+ 22us) op_driver.cc:331] REPLICATION: starting
0810 18:52:22.211232 (+ 844us) log.cc:842] Serialized 2762 byte log entry
0810 18:52:22.218289 (+ 7057us) op_driver.cc:444] REPLICATION: finished
0810 18:52:22.218740 (+ 451us) write_op.cc:301] APPLY: starting
0810 18:52:22.218821 (+ 81us) tablet.cc:1365] starting BulkCheckPresence
0810 18:52:22.218995 (+ 174us) tablet.cc:1368] finished BulkCheckPresence
0810 18:52:22.219114 (+ 119us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 18:52:22.219869 (+ 755us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 18:52:22.220059 (+ 190us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0810 18:52:22.220101 (+ 42us) write_op.cc:312] APPLY: finished
0810 18:52:22.220325 (+ 224us) log.cc:842] Serialized 53 byte log entry
0810 18:52:22.220469 (+ 144us) write_op.cc:489] Releasing partition, row and schema locks
0810 18:52:22.220578 (+ 109us) write_op.cc:454] Released schema lock
0810 18:52:22.220616 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0810 18:52:22.220849 (+ 233us) write_op.cc:489] Releasing partition, row and schema locks
0810 18:52:22.220884 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1789130,"reported_tablets":19,"child_traces":[["op",{"apply.queue_time_us":229,"apply.run_cpu_time_us":2442,"apply.run_wall_time_us":2442,"num_ops":6,"prepare.queue_time_us":124,"prepare.run_cpu_time_us":3708,"prepare.run_wall_time_us":13837,"replication_time_us":7805}]]}
I20260810 18:52:22.383450 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.380825 16624 raft_consensus.cc:2454] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
I20260810 18:52:22.376682 17783 raft_consensus.cc:2454] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
W20260810 18:52:22.399700 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 70946.5us
I20260810 18:52:22.403769 17726 leader_election.cc:302] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:22.339413 20439 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:22.406271 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.407791 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.410069 16623 raft_consensus.cc:2454] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:22.386128 20437 consensus_queue.cc:1035] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.026s
I20260810 18:52:22.413084 20439 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:22.397369 20758 consensus_queue.cc:1035] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 18:52:22.310112 20713 consensus_queue.cc:260] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:22.435738 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:22.611286 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 207582us
I20260810 18:52:22.619212 17726 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:22.544068 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:52:22.610786 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.621455 17726 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:22.613914 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 271591us
I20260810 18:52:22.624044 16575 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:22.625821 16575 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:22.628042 16575 leader_election.cc:302] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:22.629406 16575 leader_election.cc:302] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:22.631665 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:22.632656 16631 raft_consensus.cc:2396] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260810 18:52:22.633231 20459 raft_consensus.cc:2726] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:22.633906 20459 raft_consensus.cc:2726] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260810 18:52:22.643831 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 139684us
I20260810 18:52:22.400925 20683 raft_consensus.cc:686] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:22.650514 16575 leader_election.cc:302] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:22.654141 16575 leader_election.cc:302] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:22.422818 20415 raft_consensus.cc:2781] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:22.659262 16575 leader_election.cc:302] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:22.641736 20591 raft_consensus.cc:2726] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:22.641549 20373 raft_consensus.cc:2726] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260810 18:52:22.505685 16343 catalog_manager.cc:5495] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 5 to 6, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 6 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:52:22.649284 20494 raft_consensus.cc:2726] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:22.644140 20458 raft_consensus.cc:2726] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:22.671686 20373 raft_consensus.cc:2726] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:22.671442 16343 catalog_manager.cc:5495] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 4 to 5, leader changed from d304ac47fdf846de81337057a4ef7bea (127.14.32.193) to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 5 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:52:22.661279 20464 raft_consensus.cc:2726] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:22.662315 20415 raft_consensus.cc:482] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:22.680238 16343 catalog_manager.cc:5495] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 2 to 4, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 4 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } }
W20260810 18:52:22.684645 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 62257.1us
I20260810 18:52:22.688460 16343 catalog_manager.cc:5495] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 3 to 4, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 4 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:52:22.690351 16343 catalog_manager.cc:5495] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 2 to 5, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 5 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:52:22.692456 16343 catalog_manager.cc:5495] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 2 to 3, leader changed from d304ac47fdf846de81337057a4ef7bea (127.14.32.193) to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 3 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:52:22.697836 16742 raft_consensus.cc:1261] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260810 18:52:22.683427 20415 raft_consensus.cc:3037] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:22.703379 16343 catalog_manager.cc:5495] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:52:22.693792 20439 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:22.707769 16343 catalog_manager.cc:5495] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:52:22.711536 20693 consensus_queue.cc:1035] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260810 18:52:22.715147 16343 catalog_manager.cc:5495] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: config changed from index 2 to 4, 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER d304ac47fdf846de81337057a4ef7bea (127.14.32.193), VOTER 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194), VOTER 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195), VOTER 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196). New cstate: current_term: 5 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:52:22.762997 16628 raft_consensus.cc:1203] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Deduplicated request from leader. Original: 4.3->[6.4-6.4] Dedup: 6.4->[]
I20260810 18:52:22.764629 16628 raft_consensus.cc:1226] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260810 18:52:22.766010 16343 catalog_manager.cc:5495] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 4 to 5, leader changed from d304ac47fdf846de81337057a4ef7bea (127.14.32.193) to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 5 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:52:22.772056 17783 raft_consensus.cc:1261] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260810 18:52:22.776980 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 117812us
I20260810 18:52:22.777746 16343 catalog_manager.cc:5495] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260810 18:52:22.779134 16575 leader_election.cc:302] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:22.767195 20536 raft_consensus.cc:2726] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:22.789827 16701 leader_election.cc:302] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:22.714421 20683 consensus_queue.cc:237] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:22.797188 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 175826us
I20260810 18:52:22.798804 16701 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:22.800716 20771 log.cc:924] Time spent T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
W20260810 18:52:22.802241 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 109334us
I20260810 18:52:22.794214 20494 raft_consensus.cc:2726] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:22.808313 16343 catalog_manager.cc:5495] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 1 to 3, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 3 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } }
I20260810 18:52:22.780555 20464 consensus_queue.cc:1035] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20260810 18:52:22.817440 16505 raft_consensus.cc:1261] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260810 18:52:22.803915 20571 raft_consensus.cc:2726] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:22.821636 16636 raft_consensus.cc:1261] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260810 18:52:22.849112 16343 catalog_manager.cc:5495] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 1 to 2, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 2 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:52:22.859282 20689 raft_consensus.cc:2726] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:22.831553 20571 raft_consensus.cc:2726] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260810 18:52:22.861084 20764 log.cc:924] Time spent T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.144s user 0.000s sys 0.001s
W20260810 18:52:22.870075 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 71378us
I20260810 18:52:22.891716 16701 leader_election.cc:302] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:22.893426 16701 leader_election.cc:302] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:22.896695 16701 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:22.920851 16701 leader_election.cc:302] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:22.919878 16343 catalog_manager.cc:5495] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 1 to 3, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 3 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } }
I20260810 18:52:22.922955 16701 leader_election.cc:302] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:22.925478 16701 leader_election.cc:302] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:22.925603 16343 catalog_manager.cc:5495] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 1 to 3, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 3 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } }
I20260810 18:52:22.924458 20584 raft_consensus.cc:2726] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:22.928187 17781 raft_consensus.cc:1261] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260810 18:52:22.933516 16701 leader_election.cc:302] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:22.935902 16701 leader_election.cc:302] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:22.922906 20592 raft_consensus.cc:2726] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260810 18:52:22.917651 20513 raft_consensus.cc:2726] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260810 18:52:22.943337 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 85582.2us
I20260810 18:52:22.917920 20571 raft_consensus.cc:2726] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:52:22.935431 16343 catalog_manager.cc:5495] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 1 to 3, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 3 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } }
I20260810 18:52:22.947193 16343 catalog_manager.cc:5495] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 3 to 6, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 6 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } }
I20260810 18:52:22.948392 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:22.971488 16505 raft_consensus.cc:2454] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
I20260810 18:52:22.972519 16501 raft_consensus.cc:1261] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260810 18:52:22.937734 20584 raft_consensus.cc:2726] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:22.934783 20797 raft_consensus.cc:2726] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260810 18:52:22.979545 16506 raft_consensus.cc:1261] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260810 18:52:22.973351 20459 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:22.985157 16573 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:22.987401 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 127071us
I20260810 18:52:22.930147 20415 raft_consensus.cc:504] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:22.971611 20513 raft_consensus.cc:2726] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:52:22.955112 20796 raft_consensus.cc:2781] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:23.006280 20796 raft_consensus.cc:686] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:23.007674 20796 consensus_queue.cc:237] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:23.014199 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 81968.3us
I20260810 18:52:22.953738 20592 consensus_queue.cc:1035] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20260810 18:52:22.952258 20694 consensus_queue.cc:1035] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.053s
W20260810 18:52:23.026299 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 90456.1us
I20260810 18:52:22.993038 20464 raft_consensus.cc:2781] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:23.043639 16701 leader_election.cc:302] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:23.046777 16701 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:23.054170 16701 leader_election.cc:302] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:23.058359 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 60850.2us
I20260810 18:52:23.000380 20536 consensus_queue.cc:1035] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20260810 18:52:22.994531 20459 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:23.042966 20464 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:23.021240 20774 consensus_queue.cc:1035] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.037s
I20260810 18:52:23.051064 20513 raft_consensus.cc:2726] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:23.052778 17781 raft_consensus.cc:1226] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 3. Current term is 4. Ops: []
W20260810 18:52:23.115223 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 62052.6us
I20260810 18:52:23.057392 20584 raft_consensus.cc:3037] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:23.070994 20571 raft_consensus.cc:2726] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:23.057572 20797 consensus_queue.cc:1035] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.092s
W20260810 18:52:23.131502 20783 log.cc:924] Time spent T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.157s user 0.001s sys 0.000s
W20260810 18:52:23.122727 20794 log.cc:924] Time spent T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
W20260810 18:52:23.136207 20806 log.cc:924] Time spent T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.084s user 0.002s sys 0.000s
I20260810 18:52:23.141734 20459 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:23.114202 20464 raft_consensus.cc:3037] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:23.144955 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:23.154697 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:23.166230 16633 raft_consensus.cc:2454] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
I20260810 18:52:23.167716 17726 leader_election.cc:302] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:52:23.168959 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:23.169356 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:23.194368 17726 leader_election.cc:302] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:23.161967 20415 leader_election.cc:288] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:23.191825 20795 log.cc:924] Time spent T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.205s user 0.001s sys 0.000s
W20260810 18:52:23.201819 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 356076us
W20260810 18:52:23.206724 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 140459us
I20260810 18:52:23.194087 20459 raft_consensus.cc:2781] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:23.209352 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:23.206099 20717 raft_consensus.cc:2726] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:52:23.208901 20459 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:23.219385 20811 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:23.261598 16501 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23"
dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
mode: GRACEFUL
new_leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
from {username='slave'} at 127.0.0.1:51572
I20260810 18:52:23.262282 16501 raft_consensus.cc:595] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Received request to transfer leadership to TS 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:23.253753 20459 raft_consensus.cc:3037] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Advancing to term 7
W20260810 18:52:23.296214 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 57584.1us
I20260810 18:52:23.266566 20811 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:23.306442 20811 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:23.308697 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:23.309289 16501 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:23.347400 16695 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:23.366191 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 55813.2us
W20260810 18:52:23.348647 20819 log.cc:924] Time spent T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.138s user 0.001s sys 0.000s
I20260810 18:52:23.372016 16501 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad"
dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
mode: GRACEFUL
new_leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
from {username='slave'} at 127.0.0.1:51572
I20260810 18:52:23.374064 16501 raft_consensus.cc:595] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Received request to transfer leadership to TS 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:23.374461 16501 raft_consensus.cc:603] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260810 18:52:23.590801 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 213650us
I20260810 18:52:23.369939 20796 raft_consensus.cc:2781] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:23.674098 20847 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:23.620710 20796 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:23.704249 20831 log.cc:924] Time spent T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.468s user 0.000s sys 0.001s
I20260810 18:52:23.713470 17780 raft_consensus.cc:1261] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260810 18:52:23.621840 20836 raft_consensus.cc:482] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:23.717365 20584 consensus_meta.cc:287] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.573s user 0.010s sys 0.000s
I20260810 18:52:23.717868 20584 raft_consensus.cc:2726] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:23.717595 20836 raft_consensus.cc:504] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:23.718483 16749 raft_consensus.cc:2454] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
I20260810 18:52:23.719763 20836 leader_election.cc:288] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:23.720420 20836 consensus_queue.cc:1035] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260810 18:52:23.728734 20464 consensus_meta.cc:287] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.560s user 0.010s sys 0.000s
I20260810 18:52:23.729153 20464 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:23.730109 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:23.732446 16628 raft_consensus.cc:2379] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 6: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:23.734923 20464 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:23.735467 20464 raft_consensus.cc:482] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:23.737882 20843 log.cc:924] Time spent T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
W20260810 18:52:23.739351 20762 log.cc:924] Time spent T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.074s user 0.000s sys 0.001s
I20260810 18:52:23.735909 20464 raft_consensus.cc:504] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:23.743263 20464 leader_election.cc:288] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:23.673538 20840 raft_consensus.cc:482] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:23.748770 20713 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:23.749181 20840 raft_consensus.cc:504] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:23.751484 20840 leader_election.cc:288] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:23.755486 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 163714us
I20260810 18:52:23.754494 20796 raft_consensus.cc:3037] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:23.763172 20373 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:23.775550 20373 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:23.777904 20373 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:23.787696 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:23.787771 16505 raft_consensus.cc:1261] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260810 18:52:23.806241 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:23.807559 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:23.808885 16501 raft_consensus.cc:3037] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:23.699443 20847 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:23.829701 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:23.830855 16502 raft_consensus.cc:2454] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:23.832521 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:23.804570 20840 raft_consensus.cc:482] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:23.837579 20838 log.cc:924] Time spent T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.212s user 0.001s sys 0.000s
I20260810 18:52:23.814975 20373 raft_consensus.cc:482] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:23.822336 16501 raft_consensus.cc:2454] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 7.
I20260810 18:52:23.858795 16573 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:23.816004 20592 consensus_queue.cc:1035] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20260810 18:52:23.840126 20373 raft_consensus.cc:504] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:23.841512 20536 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:23.873881 20865 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:23.938884 20881 raft_consensus.cc:482] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:23.947980 20882 raft_consensus.cc:2781] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:24.042163 20886 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:24.113196 20875 raft_consensus.cc:482] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:24.114039 20875 raft_consensus.cc:504] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:24.116824 20875 leader_election.cc:288] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:23.843641 20840 raft_consensus.cc:504] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:24.152271 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 293543us
I20260810 18:52:24.068888 20882 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:24.157141 16573 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:24.158754 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 302530us
I20260810 18:52:24.139617 19587 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:24.069515 20536 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:24.162995 20882 raft_consensus.cc:3037] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:24.068830 20881 raft_consensus.cc:504] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:24.242771 20878 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:24.243492 20878 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:24.245947 20878 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:24.101123 20886 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:24.074961 20865 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:24.253159 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:24.180029 19587 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:24.234972 20797 raft_consensus.cc:482] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:24.282909 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 125888us
I20260810 18:52:24.291599 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:24.288484 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:24.292662 16749 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:24.287798 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:24.293226 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:24.293648 16636 raft_consensus.cc:2454] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:24.291290 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:24.293792 16750 raft_consensus.cc:3037] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Advancing to term 7
W20260810 18:52:24.284122 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 85621us
I20260810 18:52:24.296772 16747 raft_consensus.cc:2379] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 5: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:24.283223 20869 raft_consensus.cc:482] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:24.307600 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:52:24.311414 20867 log.cc:924] Time spent T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20260810 18:52:24.313289 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:52:24.316511 20834 log.cc:924] Time spent T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.949s user 0.009s sys 0.007s
I20260810 18:52:24.318500 20690 raft_consensus.cc:979] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea: : Instructing follower 0397d760589d40d3b92ef141cdfc2dbb to start an election
I20260810 18:52:24.320221 16626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60066 (request call id 229) took 1248 ms. Trace:
I20260810 18:52:24.320641 16626 rpcz_store.cc:276] 0810 18:52:23.072136 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:23.072336 (+ 200us) service_pool.cc:227] Handling call
0810 18:52:23.112548 (+ 40212us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:23.114166 (+ 1618us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:23.114196 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:23.114641 (+ 445us) log.cc:842] Serialized 26 byte log entry
0810 18:52:23.204758 (+ 90117us) raft_consensus.cc:1631] Marking committed up to 6
0810 18:52:23.204802 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:23.204890 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:24.315843 (+1110953us) spinlock_profiling.cc:243] Waited 7.81 ms on lock 0x7b44000232cc. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae50e4a95 00007f1ae50e54b2 00007f1ae7d8d511 00007f1ae7d88e1a 00007f1ae7d7cbee 00007f1ae7d7bf36 00007f1ae7d7bb58 00007f1ae7d85551 00007f1ae7d85509 00007f1ae7d85499
0810 18:52:24.319936 (+ 4093us) raft_consensus.cc:1671] finished
0810 18:52:24.319956 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:24.320162 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1240,"raft.queue_time_us":1636,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":86,"spinlock_wait_cycles":17964928,"thread_start_us":89851,"threads_started":1,"wal-append.queue_time_us":150701}
W20260810 18:52:24.326615 20796 consensus_meta.cc:287] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.537s user 0.007s sys 0.008s
I20260810 18:52:24.328405 20881 leader_election.cc:288] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:24.296928 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:24.297502 16701 leader_election.cc:302] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:24.333247 16631 raft_consensus.cc:2454] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:24.334271 16505 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d"
dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
mode: GRACEFUL
new_leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
from {username='slave'} at 127.0.0.1:51572
I20260810 18:52:24.332566 16750 raft_consensus.cc:2454] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 7.
I20260810 18:52:24.335028 16505 raft_consensus.cc:595] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Received request to transfer leadership to TS 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:24.336414 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:24.301532 20536 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:24.335361 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:24.335340 16701 leader_election.cc:302] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:24.337203 16743 raft_consensus.cc:2454] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
I20260810 18:52:24.258525 20886 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:24.270169 20890 raft_consensus.cc:482] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:24.351895 17781 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55586 (request call id 391) took 1421 ms. Trace:
I20260810 18:52:24.352468 17781 rpcz_store.cc:276] 0810 18:52:22.930663 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:22.936165 (+ 5502us) service_pool.cc:227] Handling call
0810 18:52:22.972836 (+ 36671us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:23.122406 (+149570us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:24.351565 (+1229159us) spinlock_profiling.cc:243] Waited 57.3 ms on lock 0x7b54006b03d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:24.351803 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":131837312}
I20260810 18:52:24.340680 16621 raft_consensus.cc:1261] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260810 18:52:24.376940 16701 leader_election.cc:302] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
W20260810 18:52:24.352571 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 93386.7us
I20260810 18:52:24.406745 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:24.423293 16575 leader_election.cc:302] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:24.425652 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:24.338210 20881 raft_consensus.cc:482] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:24.352037 20811 raft_consensus.cc:2781] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:24.404765 20373 leader_election.cc:288] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:24.420562 20164 raft_consensus.cc:1067] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Signalling peer 0397d760589d40d3b92ef141cdfc2dbb to start an election
I20260810 18:52:24.443359 16505 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e"
dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
mode: GRACEFUL
new_leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
from {username='slave'} at 127.0.0.1:51572
I20260810 18:52:24.458776 16505 raft_consensus.cc:595] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Received request to transfer leadership to TS 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:24.462576 20891 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:24.463214 20891 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:24.465534 20184 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:24.465610 20891 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:24.466219 20184 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:24.467330 16506 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56468 (request call id 250) took 1263 ms. Trace:
W20260810 18:52:24.467408 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 44227.5us
I20260810 18:52:24.468367 20184 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:24.362556 20458 raft_consensus.cc:482] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:24.470144 20458 raft_consensus.cc:504] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:24.470471 20894 raft_consensus.cc:2781] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:24.471022 20894 raft_consensus.cc:482] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:24.471470 20894 raft_consensus.cc:3037] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:24.403612 20584 raft_consensus.cc:2726] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:24.471814 20886 raft_consensus.cc:482] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:24.430712 20901 raft_consensus.cc:2781] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:24.472411 20901 raft_consensus.cc:482] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:24.472329 20886 raft_consensus.cc:504] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:24.472899 20901 raft_consensus.cc:3037] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:24.474488 20886 leader_election.cc:288] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:24.447155 20874 log.cc:924] Time spent T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.178s user 0.001s sys 0.000s
I20260810 18:52:24.431583 20861 consensus_queue.cc:1035] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.017s
I20260810 18:52:24.476416 20903 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:24.476815 20458 leader_election.cc:288] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:24.272202 20797 raft_consensus.cc:504] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:24.449862 20882 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:24.476943 20903 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:24.478778 20882 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:24.470667 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:24.467775 16506 rpcz_store.cc:276] 0810 18:52:23.204451 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:23.204844 (+ 393us) service_pool.cc:227] Handling call
0810 18:52:23.218995 (+ 14151us) raft_consensus.cc:1457] Updating replica for 2 ops
0810 18:52:23.223671 (+ 4676us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:23.223702 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0810 18:52:23.254215 (+ 30513us) log.cc:842] Serialized 52 byte log entry
0810 18:52:23.574578 (+320363us) raft_consensus.cc:1631] Marking committed up to 7
0810 18:52:23.622338 (+ 47760us) log.cc:842] Serialized 14 byte log entry
0810 18:52:23.622736 (+ 398us) log.cc:842] Serialized 14 byte log entry
0810 18:52:23.622912 (+ 176us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:23.623008 (+ 96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:24.309096 (+686088us) spinlock_profiling.cc:243] Waited 7.34 ms on lock 0x7b1800479680. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1aee0718b1 00007f1aee07e661 00007f1ae51d2638 00007f1ae51d5b11 00007f1ae51d5ac9 00007f1ae51d5a59 00007f1ae51d5a21 00007f1ae51d4d1d 00007f1aee436204 00007f1aee436039 00007f1ae51b4dc6
0810 18:52:24.464601 (+155505us) spinlock_profiling.cc:243] Waited 49.8 ms on lock 0x7b1800479680. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1aee0718b1 00007f1aee07e661 00007f1ae51d2638 00007f1ae51d5b11 00007f1ae51d5ac9 00007f1ae51d5a59 00007f1ae51d5a21 00007f1ae51d4d1d 00007f1aee436204 00007f1aee436039 00007f1ae51b4dc6
0810 18:52:24.467119 (+ 2518us) raft_consensus.cc:1671] finished
0810 18:52:24.467143 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:24.467273 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33700,"prepare.queue_time_us":178954,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":5032,"raft.queue_time_us":182820,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":252,"spinlock_wait_cycles":131517568,"thread_start_us":276031,"threads_started":1,"wal-append.queue_time_us":436474}
I20260810 18:52:24.480537 20903 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:24.405570 20890 raft_consensus.cc:504] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:24.479912 20865 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:24.482720 20890 leader_election.cc:288] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:24.307721 20869 raft_consensus.cc:504] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:24.485090 20869 leader_election.cc:288] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:24.490756 16505 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712"
dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
mode: GRACEFUL
new_leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
from {username='slave'} at 127.0.0.1:51572
I20260810 18:52:24.491533 16505 raft_consensus.cc:595] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Received request to transfer leadership to TS 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:24.492798 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260810 18:52:24.350127 20796 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:24.495407 20796 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:24.478778 20797 leader_election.cc:288] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:24.498322 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:24.492700 20847 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:24.416074 20694 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
W20260810 18:52:24.844769 16447 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.637s user 0.019s sys 0.060s
I20260810 18:52:24.953402 20885 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:24.905800 16626 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.003s sys 0.029s
I20260810 18:52:24.434441 20758 raft_consensus.cc:2781] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader election won for term 7
W20260810 18:52:24.924254 16695 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.840s user 0.025s sys 0.033s
I20260810 18:52:24.539444 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:24.432263 20536 raft_consensus.cc:482] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:24.435011 20859 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:24.500463 20881 raft_consensus.cc:504] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:24.973136 16626 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.642s user 0.006s sys 0.036s
I20260810 18:52:24.499423 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:24.619139 17783 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.549s user 0.012s sys 0.042s
W20260810 18:52:24.969115 16573 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.554s user 0.015s sys 0.027s
W20260810 18:52:25.002061 17783 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.932s user 0.033s sys 0.063s
W20260810 18:52:25.002113 16573 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.587s user 0.015s sys 0.028s
I20260810 18:52:24.506191 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:24.498703 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:24.973114 16447 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.765s user 0.025s sys 0.072s
I20260810 18:52:24.506790 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:24.499807 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:52:25.003844 16636 raft_consensus.cc:2379] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 6: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:24.438027 20840 leader_election.cc:288] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:24.616480 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 126669us
I20260810 18:52:25.004184 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:52:25.003175 16633 raft_consensus.cc:2454] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 2.
W20260810 18:52:25.002759 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 639626us
I20260810 18:52:25.007555 16636 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23"
dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
from {username='slave'} at 127.0.0.1:53870
W20260810 18:52:25.008037 16695 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.924s user 0.027s sys 0.038s
I20260810 18:52:25.008981 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:25.002117 16745 raft_consensus.cc:2454] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:25.009215 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:25.016211 16740 raft_consensus.cc:2454] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
W20260810 18:52:25.017153 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 933572us
I20260810 18:52:24.444502 20811 raft_consensus.cc:482] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:25.012563 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:24.981755 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:25.010097 16621 raft_consensus.cc:2379] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 6: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:25.018327 16752 raft_consensus.cc:2379] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 5: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:25.014729 16636 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:52:25.021239 16636 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Advancing to term 8
W20260810 18:52:24.863875 17771 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.031s sys 0.045s
I20260810 18:52:25.033046 16573 leader_election.cc:302] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:52:25.046007 17771 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.734s user 0.035s sys 0.061s
I20260810 18:52:25.001463 20886 raft_consensus.cc:482] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:24.937662 20861 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:24.913489 20415 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:25.004879 20694 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:25.004792 20536 raft_consensus.cc:504] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:52:25.031615 17780 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55602 (request call id 381) took 1140 ms. Trace:
I20260810 18:52:24.662257 20571 raft_consensus.cc:482] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:24.698338 20711 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:25.031718 20811 raft_consensus.cc:3037] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:24.660992 20796 raft_consensus.cc:482] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:24.917371 20584 raft_consensus.cc:482] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:24.615484 20797 raft_consensus.cc:482] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:24.662676 20561 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:24.934842 20683 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:24.637202 20319 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
W20260810 18:52:25.034880 20844 log.cc:924] Time spent T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20260810 18:52:25.033152 20840 raft_consensus.cc:482] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:25.076118 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:25.098135 20881 leader_election.cc:288] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:25.098704 20694 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:25.090510 17780 rpcz_store.cc:276] 0810 18:52:23.872220 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:24.051359 (+179139us) service_pool.cc:227] Handling call
0810 18:52:24.148827 (+ 97468us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:24.164777 (+ 15950us) raft_consensus.cc:1520] Early marking committed up to index 7
0810 18:52:24.660658 (+495881us) log.cc:842] Serialized 14 byte log entry
0810 18:52:24.845195 (+184537us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:24.918752 (+ 73557us) raft_consensus.cc:1631] Marking committed up to 7
0810 18:52:24.948644 (+ 29892us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:24.977242 (+ 28598us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:25.012933 (+ 35691us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3296,"prepare.queue_time_us":774355,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":139452,"raft.run_cpu_time_us":5688,"raft.run_wall_time_us":59929,"thread_start_us":389589,"threads_started":1}
I20260810 18:52:25.012380 20859 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:25.103258 20711 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:25.105263 20536 leader_election.cc:288] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:25.091867 20415 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:25.043656 20869 raft_consensus.cc:482] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:25.053747 19587 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:25.010031 20885 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:25.094280 20886 raft_consensus.cc:504] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.064654 20903 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:25.065224 20459 consensus_meta.cc:287] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 1.754s user 0.012s sys 0.008s
I20260810 18:52:25.114800 16636 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.154278 16636 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:25.122749 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:25.130543 20881 raft_consensus.cc:482] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:25.059778 20890 raft_consensus.cc:482] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:25.079209 20912 raft_consensus.cc:482] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:25.116895 20847 raft_consensus.cc:482] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:25.141186 20536 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:25.144014 20711 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:25.144258 20415 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:25.128886 20840 raft_consensus.cc:504] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:25.164907 20713 consensus_meta.cc:287] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 1.413s user 0.011s sys 0.008s
I20260810 18:52:25.124778 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:25.127454 20683 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.091696 20861 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.119279 20796 raft_consensus.cc:504] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:25.155181 20901 consensus_meta.cc:287] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.682s user 0.009s sys 0.003s
I20260810 18:52:25.118875 20571 raft_consensus.cc:504] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.163117 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:25.121006 20584 raft_consensus.cc:504] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.123299 20797 raft_consensus.cc:504] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.126705 20561 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.141389 20869 raft_consensus.cc:504] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.155638 16748 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
I20260810 18:52:25.126557 20319 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:25.160792 17770 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:25.157254 20459 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:25.156914 20903 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:25.164299 20881 raft_consensus.cc:504] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.173383 20247 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:25.162676 20890 raft_consensus.cc:504] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.255105 16750 raft_consensus.cc:2454] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 7.
W20260810 18:52:25.179723 20894 consensus_meta.cc:287] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.707s user 0.008s sys 0.003s
I20260810 18:52:25.167577 20847 raft_consensus.cc:504] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:52:25.237831 16701 leader_election.cc:302] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:25.185240 20886 leader_election.cc:288] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:25.165336 20912 raft_consensus.cc:504] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.169893 20536 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.265841 17778 raft_consensus.cc:1226] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:25.252385 20694 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:25.253530 20717 raft_consensus.cc:482] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:25.209239 20907 raft_consensus.cc:482] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:25.180567 20901 raft_consensus.cc:504] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.281343 20717 raft_consensus.cc:504] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:25.282222 20912 leader_election.cc:288] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:25.260222 20914 raft_consensus.cc:2726] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:25.270821 20886 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:25.293005 20912 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:25.257994 20247 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:25.258996 20894 raft_consensus.cc:504] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.300853 20912 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.306527 20885 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:25.285506 20907 raft_consensus.cc:504] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.331322 20592 raft_consensus.cc:482] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:25.331945 20592 raft_consensus.cc:504] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.332974 20907 leader_election.cc:288] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:25.280026 20811 raft_consensus.cc:504] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.333992 20592 leader_election.cc:288] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:25.334822 20811 leader_election.cc:288] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:25.335433 20811 raft_consensus.cc:2781] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:52:25.335757 20811 raft_consensus.cc:482] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:25.336149 20811 raft_consensus.cc:3037] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:25.336431 20912 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:25.308048 20905 log.cc:924] Time spent T 00000000000000000000000000000000 P 0975c6fa18274f2baee7fb3615d5b985: Append to log took a long time: real 0.128s user 0.002s sys 0.000s
I20260810 18:52:25.307834 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:25.339259 16632 raft_consensus.cc:2379] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 6: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
W20260810 18:52:25.309124 20910 log.cc:924] Time spent T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.137s user 0.000s sys 0.001s
I20260810 18:52:25.310972 20717 leader_election.cc:288] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:25.311807 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:25.325170 20847 leader_election.cc:288] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:25.349647 16507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:52330 (request call id 832) took 3033 ms. Trace:
I20260810 18:52:25.281039 20694 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:25.353719 16623 raft_consensus.cc:1226] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:25.350126 16507 rpcz_store.cc:276] 0810 18:52:22.308111 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:22.308846 (+ 735us) service_pool.cc:227] Handling call
0810 18:52:25.340455 (+3031609us) spinlock_profiling.cc:243] Waited 2.86 s on lock 0x7b54000f7154. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807faf0 00007f1ae8102f2a 00007f1aed03fc6e 00007f1ae6736807 00007f1ae6736792 00007f1ae67366c1 00007f1ae673663c 00007f1ae67358f2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:25.341230 (+ 775us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6584306560}
I20260810 18:52:25.302812 20859 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:25.352854 20839 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:25.334957 20592 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:25.358548 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 325599us
I20260810 18:52:25.359249 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:25.362393 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:25.336508 20861 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:25.299464 20886 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:25.377616 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:25.375954 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:25.395349 20903 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:25.395701 20891 raft_consensus.cc:482] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:25.396289 20891 raft_consensus.cc:504] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.396757 20319 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:25.397503 20571 leader_election.cc:288] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:25.398206 20549 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:25.398500 20891 leader_election.cc:288] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:25.398828 20549 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.400476 20694 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:25.401140 20869 leader_election.cc:288] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:25.401381 20549 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:25.401942 20890 leader_election.cc:288] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:25.395426 20561 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:25.403184 17783 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55586 (request call id 395) took 1552 ms. Trace:
I20260810 18:52:25.403396 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:25.404644 20861 raft_consensus.cc:482] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:25.403690 17783 rpcz_store.cc:276] 0810 18:52:23.850576 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:23.850732 (+ 156us) service_pool.cc:227] Handling call
0810 18:52:23.851225 (+ 493us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:23.851678 (+ 453us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:23.851722 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:23.862976 (+ 11254us) log.cc:842] Serialized 26 byte log entry
0810 18:52:25.002560 (+1139584us) raft_consensus.cc:1631] Marking committed up to 6
0810 18:52:25.002598 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:25.002704 (+ 106us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:25.308486 (+305782us) spinlock_profiling.cc:243] Waited 3.27 ms on lock 0x7b44000232cc. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae50e4a95 00007f1ae50e54b2 00007f1ae7d8d511 00007f1ae7d88e1a 00007f1ae7d7cbee 00007f1ae7d7bf36 00007f1ae7d7bb58 00007f1ae7d85551 00007f1ae7d85509 00007f1ae7d85499
0810 18:52:25.402773 (+ 94287us) raft_consensus.cc:1671] finished
0810 18:52:25.402804 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:25.403106 (+ 302us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1348,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":15178,"spinlock_wait_cycles":7525760,"thread_start_us":932035,"threads_started":1,"wal-append.queue_time_us":1292677}
I20260810 18:52:25.405157 20861 raft_consensus.cc:504] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.360103 20881 leader_election.cc:288] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:25.358093 20796 leader_election.cc:288] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:25.357764 20717 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
W20260810 18:52:25.441708 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 53001.9us
I20260810 18:52:25.443948 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:25.364784 20840 leader_election.cc:288] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:25.360061 20536 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:25.368068 20912 raft_consensus.cc:482] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:25.452733 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:25.449677 16626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60066 (request call id 239) took 1119 ms. Trace:
I20260810 18:52:25.360699 20459 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:25.362092 20584 leader_election.cc:288] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:25.440050 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 427980us
I20260810 18:52:25.361785 20683 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:25.367871 20847 raft_consensus.cc:482] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:25.360486 20797 leader_election.cc:288] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:25.454819 16626 rpcz_store.cc:276] 0810 18:52:24.330400 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:24.330607 (+ 207us) service_pool.cc:227] Handling call
0810 18:52:24.330920 (+ 313us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:24.331119 (+ 199us) raft_consensus.cc:1520] Early marking committed up to index 8
0810 18:52:25.001765 (+670646us) log.cc:842] Serialized 14 byte log entry
0810 18:52:25.449369 (+447604us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:25.449423 (+ 54us) raft_consensus.cc:1631] Marking committed up to 8
0810 18:52:25.449466 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:25.449530 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:25.449625 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":51152,"prepare.queue_time_us":813008,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":60,"raft.queue_time_us":166780,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":7859,"thread_start_us":1066366,"threads_started":2}
I20260810 18:52:25.457274 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:25.370013 20164 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:25.403424 20901 leader_election.cc:288] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:25.461503 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:25.369810 20839 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:25.407471 20914 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:25.377796 20859 raft_consensus.cc:482] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:25.368415 20592 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.468533 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:25.439757 20890 raft_consensus.cc:482] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:25.468366 16633 raft_consensus.cc:2454] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 5.
W20260810 18:52:25.470719 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 375251us
I20260810 18:52:25.607055 17726 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:52:25.607295 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:25.607800 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:25.608403 16507 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
I20260810 18:52:25.609828 16742 raft_consensus.cc:1226] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:25.553695 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:25.609855 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:25.450448 20717 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:25.454377 20891 raft_consensus.cc:482] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:25.451355 19587 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
W20260810 18:52:25.659497 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 43759.5us
I20260810 18:52:25.643188 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:25.661407 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:25.659897 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:25.553296 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 80459.5us
I20260810 18:52:25.471988 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:25.705039 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:25.606843 20758 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.005s sys 0.048s
I20260810 18:52:25.546226 20903 raft_consensus.cc:482] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:25.458141 20796 consensus_queue.cc:1046] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.275s
I20260810 18:52:25.451189 20319 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:25.573971 17771 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55602 (request call id 383) took 1242 ms. Trace:
I20260810 18:52:25.740181 16495 raft_consensus.cc:2454] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:25.451750 20713 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:25.706698 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:25.793723 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 77267.2us
W20260810 18:52:25.734648 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 71858.5us
I20260810 18:52:25.471235 20858 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:25.577848 20813 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:25.611413 20861 leader_election.cc:288] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:25.579586 20869 raft_consensus.cc:482] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:25.606685 20907 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:25.574290 20802 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:25.545037 20536 raft_consensus.cc:482] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:25.800256 20758 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.738s user 0.018s sys 0.068s
I20260810 18:52:25.822156 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:25.804677 17778 raft_consensus.cc:2454] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:25.825493 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:25.839046 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:25.839690 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:25.933485 16496 raft_consensus.cc:3032] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Stepping down as leader of term 7
I20260810 18:52:25.839862 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:25.790079 20894 leader_election.cc:288] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:25.851516 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:25.469619 20847 raft_consensus.cc:504] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.874626 16631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1560) took 1373 ms. Trace:
I20260810 18:52:25.699251 19587 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:25.699602 20694 raft_consensus.cc:2781] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260810 18:52:25.914105 16573 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.14.32.193:40791 timed out after 1.555s (SENT) [suppressed 15 similar messages]
I20260810 18:52:25.791823 20885 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:25.707085 20891 raft_consensus.cc:504] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.699741 20920 raft_consensus.cc:482] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:25.887724 20886 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:25.884968 19996 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:25.821290 20319 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:25.927852 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 320874us
I20260810 18:52:25.933017 16636 raft_consensus.cc:2454] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:25.933885 16496 raft_consensus.cc:729] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:25.458592 20912 raft_consensus.cc:504] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.883212 20606 raft_consensus.cc:979] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea: : Instructing follower 0397d760589d40d3b92ef141cdfc2dbb to start an election
I20260810 18:52:25.960645 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:25.937584 16701 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.678s user 0.074s sys 0.091s
I20260810 18:52:25.544430 20164 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:25.622316 20890 raft_consensus.cc:504] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.642740 20859 raft_consensus.cc:504] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.970839 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:25.942875 16631 rpcz_store.cc:276] 0810 18:52:24.500803 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:24.500955 (+ 152us) service_pool.cc:227] Handling call
0810 18:52:24.975466 (+474511us) raft_consensus.cc:1457] Updating replica for 2 ops
0810 18:52:25.032718 (+ 57252us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:25.032747 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0810 18:52:25.044412 (+ 11665us) log.cc:842] Serialized 52 byte log entry
0810 18:52:25.456444 (+412032us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:25.545071 (+ 88627us) log.cc:842] Serialized 14 byte log entry
0810 18:52:25.575473 (+ 30402us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:25.575589 (+ 116us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:25.874325 (+298736us) raft_consensus.cc:1671] finished
0810 18:52:25.874358 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:25.874552 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":86935,"prepare.queue_time_us":96925,"prepare.run_cpu_time_us":130,"prepare.run_wall_time_us":7936,"raft.queue_time_us":228532,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":84170,"spinlock_wait_cycles":8608640,"thread_start_us":382737,"threads_started":1,"wal-append.queue_time_us":682077}
I20260810 18:52:25.887310 20592 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:26.006592 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:25.974216 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 363111us
I20260810 18:52:25.888717 20717 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:25.622558 20914 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.891564 20839 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:25.978222 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:25.818341 20903 raft_consensus.cc:504] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.927312 20933 raft_consensus.cc:482] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:25.876138 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:25.981633 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:25.987461 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:25.938517 20758 raft_consensus.cc:686] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:25.958033 20694 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:25.991839 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:25.881116 20853 consensus_queue.cc:1046] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.097s
I20260810 18:52:25.830642 20858 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:26.049685 20858 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:25.823748 20713 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:25.885474 20840 consensus_queue.cc:1046] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.098s
W20260810 18:52:25.997400 16701 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.738s user 0.074s sys 0.092s
W20260810 18:52:26.050737 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 812947us
I20260810 18:52:26.051230 16507 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
I20260810 18:52:25.841610 20813 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:26.052004 20459 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:26.006615 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:25.871794 20927 raft_consensus.cc:482] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:26.007716 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:26.015445 20606 raft_consensus.cc:1067] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Signalling peer 0397d760589d40d3b92ef141cdfc2dbb to start an election
I20260810 18:52:25.943295 20869 raft_consensus.cc:504] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.943622 20907 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.958847 20536 raft_consensus.cc:504] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.943476 20802 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:25.973642 20885 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:26.045539 20811 consensus_meta.cc:287] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.709s user 0.012s sys 0.000s
I20260810 18:52:25.991263 19996 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:26.051664 20713 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:26.053475 16501 raft_consensus.cc:2454] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 5.
I20260810 18:52:25.986652 16496 consensus_queue.cc:260] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:26.092366 16496 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:26.052510 20459 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:26.015816 20940 raft_consensus.cc:482] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:26.016359 20945 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:25.980717 20920 raft_consensus.cc:504] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:26.042624 20694 raft_consensus.cc:3037] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:26.066319 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:26.071825 16575 leader_election.cc:302] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:26.116401 16632 raft_consensus.cc:2454] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
W20260810 18:52:26.117043 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 45316.7us
I20260810 18:52:26.117689 16575 leader_election.cc:302] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
W20260810 18:52:26.115020 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 103667us
I20260810 18:52:26.119017 16575 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:52:26.095182 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 181115us
I20260810 18:52:26.104455 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:26.122330 16508 raft_consensus.cc:2454] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
I20260810 18:52:26.105719 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:26.124131 16501 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
I20260810 18:52:26.125689 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:26.126556 16623 raft_consensus.cc:2454] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
I20260810 18:52:26.106163 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:26.127436 16740 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:26.107525 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:26.089706 20942 raft_consensus.cc:3032] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Stepping down as leader of term 6
I20260810 18:52:26.110934 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:26.112202 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:26.129938 16575 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:26.130621 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:26.109772 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 109018us
I20260810 18:52:26.132774 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:26.134474 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:26.135260 16747 raft_consensus.cc:2454] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:26.142005 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:26.139765 16624 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
I20260810 18:52:26.148746 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:26.149546 16501 raft_consensus.cc:2454] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
I20260810 18:52:26.104240 20840 raft_consensus.cc:482] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:26.113655 20861 raft_consensus.cc:482] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:26.114115 20813 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:26.040745 20933 raft_consensus.cc:504] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:26.162528 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:26.134825 20942 raft_consensus.cc:729] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:26.092291 20811 raft_consensus.cc:504] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:26.136269 20886 raft_consensus.cc:2781] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:26.133873 20894 raft_consensus.cc:2726] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260810 18:52:26.323869 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 194017us
I20260810 18:52:26.179613 20847 leader_election.cc:288] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:26.143826 20713 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:26.063010 20927 raft_consensus.cc:504] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:25.806188 17771 rpcz_store.cc:276] 0810 18:52:24.310513 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:24.310702 (+ 189us) service_pool.cc:227] Handling call
0810 18:52:24.311176 (+ 474us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:24.311490 (+ 314us) raft_consensus.cc:1520] Early marking committed up to index 8
0810 18:52:25.046474 (+734984us) log.cc:842] Serialized 14 byte log entry
0810 18:52:25.454298 (+407824us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:25.462214 (+ 7916us) raft_consensus.cc:1631] Marking committed up to 8
0810 18:52:25.466540 (+ 4326us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:25.471194 (+ 4654us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:25.553123 (+ 81929us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":760940,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":1128,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":7371,"thread_start_us":1129939,"threads_started":2,"wal-append.queue_time_us":775844}
I20260810 18:52:26.135180 20937 raft_consensus.cc:2781] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:26.386526 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:26.431897 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:26.385510 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:26.398715 16448 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:26.432569 16624 raft_consensus.cc:2454] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:26.433435 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:26.398825 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:26.332773 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:26.386204 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:26.364347 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:26.433921 16750 raft_consensus.cc:1226] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 2. Current term is 3. Ops: []
I20260810 18:52:26.436793 16631 raft_consensus.cc:2454] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:26.437620 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:26.438181 16631 raft_consensus.cc:2454] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 1.
I20260810 18:52:26.438907 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:26.444602 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 45996.6us
I20260810 18:52:26.449893 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:26.450452 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:26.451282 16628 raft_consensus.cc:2454] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:26.451457 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:26.452308 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:26.452893 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:26.453646 16623 raft_consensus.cc:2454] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:26.454605 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:26.453646 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:26.179103 20840 raft_consensus.cc:504] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:26.277271 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:26.114739 20940 raft_consensus.cc:504] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:26.220856 20861 raft_consensus.cc:504] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:26.116308 20945 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:26.127219 20758 consensus_queue.cc:237] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:26.431528 20319 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:26.537894 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:26.634439 20975 raft_consensus.cc:482] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:26.635130 20975 raft_consensus.cc:504] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:26.465029 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:26.575940 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 126712us
I20260810 18:52:26.462798 17726 leader_election.cc:302] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:52:26.450416 20713 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:26.517002 16631 raft_consensus.cc:2454] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:26.654989 16628 raft_consensus.cc:2379] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 4: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
W20260810 18:52:26.590358 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 159228us
I20260810 18:52:26.284708 20886 raft_consensus.cc:482] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:26.466758 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
W20260810 18:52:26.683990 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 539815us
I20260810 18:52:26.688779 16742 raft_consensus.cc:2379] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 4: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:26.381198 20973 raft_consensus.cc:482] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:26.330056 20164 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:26.699573 16448 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:26.800812 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 101327us
I20260810 18:52:26.800990 20825 raft_consensus.cc:2781] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:26.801461 20825 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:26.431559 20937 raft_consensus.cc:482] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:26.683811 19587 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:26.776974 20920 leader_election.cc:288] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:26.431532 20912 leader_election.cc:288] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:26.783556 16507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52338 (request call id 1502) took 1169 ms. Trace:
I20260810 18:52:26.398052 20942 consensus_queue.cc:260] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:26.628647 20914 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:26.787921 16505 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1498) took 1316 ms. Trace:
W20260810 18:52:26.791447 16740 consensus_meta.cc:287] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.663s user 0.002s sys 0.010s
I20260810 18:52:26.401619 20894 raft_consensus.cc:2781] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:26.792510 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:26.792778 16447 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:26.539891 20891 leader_election.cc:288] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:26.461643 19907 raft_consensus.cc:2781] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:26.681309 20903 leader_election.cc:288] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:26.836457 20937 raft_consensus.cc:3037] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:26.656895 20927 leader_election.cc:288] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:26.681255 19996 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:26.686149 20859 leader_election.cc:288] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:26.683127 20901 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:26.683079 20886 raft_consensus.cc:3037] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:26.689061 20869 leader_election.cc:288] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:26.788832 20918 raft_consensus.cc:3032] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:52:26.847434 16507 rpcz_store.cc:276] 0810 18:52:25.614263 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:25.615364 (+ 1101us) service_pool.cc:227] Handling call
0810 18:52:26.773587 (+1158223us) spinlock_profiling.cc:243] Waited 389 ms on lock 0x7b54001388d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:26.783487 (+ 9900us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":894709632}
I20260810 18:52:26.884944 20918 raft_consensus.cc:729] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:26.848471 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:26.885250 20974 log.cc:924] Time spent T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
I20260810 18:52:26.885874 20918 consensus_queue.cc:260] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:26.887010 20713 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:26.887151 20901 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:26.887725 20861 leader_election.cc:288] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:26.888230 20861 raft_consensus.cc:482] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:26.888664 20861 raft_consensus.cc:504] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:26.889482 20901 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:26.882309 20859 raft_consensus.cc:482] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:26.890534 20861 leader_election.cc:288] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:26.890933 20859 raft_consensus.cc:504] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:26.891813 20959 log.cc:924] Time spent T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.215s user 0.000s sys 0.000s
I20260810 18:52:26.893045 20859 leader_election.cc:288] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:26.886924 20918 raft_consensus.cc:3037] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:26.893740 20903 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:26.894296 20903 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:26.882462 20811 leader_election.cc:288] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:26.867764 20247 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:26.895645 20927 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:26.896036 20979 raft_consensus.cc:482] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:26.896341 20903 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:26.896207 20927 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:26.896625 20979 raft_consensus.cc:504] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:26.898454 20927 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:26.898757 20979 leader_election.cc:288] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:26.858361 16740 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 8.
I20260810 18:52:26.848510 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:26.900770 16628 raft_consensus.cc:2454] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:26.858805 16743 raft_consensus.cc:2454] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
W20260810 18:52:26.859213 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 66511.3us
I20260810 18:52:26.848673 16575 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:26.802826 20813 raft_consensus.cc:2781] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:26.810824 20825 raft_consensus.cc:3037] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:26.858179 16505 rpcz_store.cc:276] 0810 18:52:25.472913 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:25.484757 (+ 11844us) service_pool.cc:227] Handling call
0810 18:52:25.485117 (+ 360us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:25.485345 (+ 228us) raft_consensus.cc:1520] Early marking committed up to index 7
0810 18:52:25.970512 (+485167us) log.cc:842] Serialized 14 byte log entry
0810 18:52:26.772052 (+801540us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b180008d5e0. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1aee0718b1 00007f1aee07e661 00007f1ae7d8982c 00007f1ae51b49dc 00007f1aedfeba39 00007f1ae51d18bb 00007f1ae51cfcbf 00007f1ae51d1e22 00007f1ae7d7b5bb 00007f1ae7d81063 00007f1ae7d8157b
0810 18:52:26.772301 (+ 249us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:26.787549 (+ 15248us) raft_consensus.cc:1631] Marking committed up to 7
0810 18:52:26.787611 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:26.787700 (+ 89us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:26.787852 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":573528,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":49,"raft.queue_time_us":2791,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":114,"spinlock_wait_cycles":37100288,"thread_start_us":500290,"threads_started":2,"wal-append.queue_time_us":568828}
I20260810 18:52:26.810052 20536 leader_election.cc:288] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:26.802282 16448 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:26.855191 20943 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.675s user 0.046s sys 0.084s
I20260810 18:52:26.832728 20934 raft_consensus.cc:482] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:26.813807 20797 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:26.840373 20885 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:26.861075 20894 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:26.861889 20907 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:26.877214 20891 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:26.872179 20914 raft_consensus.cc:482] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:26.923942 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:26.924373 20914 raft_consensus.cc:504] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:26.926448 16746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1959) took 1336 ms. Trace:
I20260810 18:52:26.864998 19587 raft_consensus.cc:2781] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:26.865193 19907 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:26.872126 20978 raft_consensus.cc:482] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:26.866516 20459 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:26.919595 20943 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.806s user 0.046s sys 0.085s
W20260810 18:52:26.859941 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 397237us
W20260810 18:52:26.916599 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 114385us
I20260810 18:52:26.917202 20898 raft_consensus.cc:2781] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:26.907454 20979 raft_consensus.cc:2781] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:26.905406 20813 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:26.867443 20802 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:26.923195 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:26.926842 20914 leader_election.cc:288] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:26.923295 20891 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:26.902964 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 54398.4us
I20260810 18:52:26.920809 20894 raft_consensus.cc:3037] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:26.981155 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:26.933441 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:26.983232 16575 leader_election.cc:302] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:26.951773 19587 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:26.952492 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:26.988739 16740 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 5.
I20260810 18:52:26.989722 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:26.954337 16573 leader_election.cc:302] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:26.906781 16496 consensus_meta.cc:287] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.814s user 0.007s sys 0.004s
I20260810 18:52:26.990927 16496 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 8.
I20260810 18:52:26.991777 16496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52338 (request call id 1506) took 1248 ms. Trace:
I20260810 18:52:26.800797 20973 raft_consensus.cc:504] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:26.976068 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:26.999830 16745 raft_consensus.cc:2454] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:27.000451 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:27.000656 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:26.977097 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:26.977849 16631 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d"
dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
from {username='slave'} at 127.0.0.1:53870
I20260810 18:52:27.001657 16631 raft_consensus.cc:482] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:52:27.002173 16631 raft_consensus.cc:3037] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:26.953096 19907 raft_consensus.cc:3037] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:26.926784 16746 rpcz_store.cc:276] 0810 18:52:25.590417 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:26.925741 (+1335324us) service_pool.cc:227] Handling call
0810 18:52:26.926022 (+ 281us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:26.926181 (+ 159us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:26.926205 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:26.926240 (+ 35us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:26.926267 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:26.926311 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:26.926397 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:27.007032 16624 raft_consensus.cc:2379] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 8: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:26.928061 16749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1955) took 1436 ms. Trace:
I20260810 18:52:26.973630 20898 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:27.009097 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:26.992874 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:26.980655 20813 raft_consensus.cc:3037] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:27.009841 16752 raft_consensus.cc:2454] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
I20260810 18:52:26.964711 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:26.993904 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:27.011046 16632 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
I20260810 18:52:26.990402 16628 raft_consensus.cc:2454] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:26.994110 16496 rpcz_store.cc:276] 0810 18:52:25.743336 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:25.743516 (+ 180us) service_pool.cc:227] Handling call
0810 18:52:26.991703 (+1248187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:26.995787 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:26.998540 16701 leader_election.cc:302] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:27.011965 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:27.012106 16748 raft_consensus.cc:2454] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:52:27.001600 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:27.001157 16745 raft_consensus.cc:2454] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:52:27.013409 16740 raft_consensus.cc:2454] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:27.013688 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:26.972743 20979 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:27.015980 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:26.972821 20459 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:26.965142 20975 leader_election.cc:288] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:26.986774 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:27.007428 16749 rpcz_store.cc:276] 0810 18:52:25.491821 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:26.927284 (+1435463us) service_pool.cc:227] Handling call
0810 18:52:26.927613 (+ 329us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:26.927766 (+ 153us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:26.927790 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:26.927828 (+ 38us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:26.927855 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:26.927912 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:26.928009 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:26.935860 20797 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:27.001133 16621 raft_consensus.cc:2454] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:26.987315 17778 raft_consensus.cc:2379] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 7: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:52:26.919037 20934 raft_consensus.cc:504] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:27.004067 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 49818.7us
I20260810 18:52:26.987008 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:26.964314 20978 raft_consensus.cc:504] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:27.006161 20840 leader_election.cc:288] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:27.293818 20813 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:27.017158 20898 raft_consensus.cc:3037] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:27.071137 20994 raft_consensus.cc:2781] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:27.313905 20994 raft_consensus.cc:482] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:27.314285 20994 raft_consensus.cc:3037] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:27.282054 20937 raft_consensus.cc:504] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:27.317081 20937 leader_election.cc:288] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:27.286918 20825 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:27.319926 20825 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:27.155022 20952 raft_consensus.cc:3032] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Stepping down as leader of term 3
W20260810 18:52:27.197261 20694 consensus_meta.cc:287] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 1.054s user 0.012s sys 0.000s
I20260810 18:52:27.321084 20952 raft_consensus.cc:729] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
W20260810 18:52:27.150625 20976 log.cc:924] Time spent T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.270s user 0.001s sys 0.000s
I20260810 18:52:27.321211 20940 leader_election.cc:288] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:27.321983 20694 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:27.314922 20813 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:27.088181 20968 log.cc:924] Time spent T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.167s user 0.000s sys 0.000s
I20260810 18:52:27.081707 20973 leader_election.cc:288] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:27.080000 20891 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:27.338925 20934 leader_election.cc:288] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:27.019218 20979 raft_consensus.cc:3037] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:27.322014 20952 consensus_queue.cc:260] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:27.009744 20927 raft_consensus.cc:2781] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader election won for term 8
I20260810 18:52:27.085282 20886 raft_consensus.cc:504] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:27.323944 20933 leader_election.cc:288] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:27.200344 20894 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:27.312521 20459 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:27.327116 20994 raft_consensus.cc:504] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:27.333103 20694 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:27.336321 20797 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:27.322382 20978 leader_election.cc:288] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:27.432757 20894 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:27.432904 20886 leader_election.cc:288] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:27.407204 20952 raft_consensus.cc:3037] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:27.078853 20945 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:27.408659 20927 raft_consensus.cc:686] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:28.141638 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:28.142598 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.142910 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:27.438920 20898 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:28.143163 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.148624 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.143429 16507 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:28.149626 20898 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:27.620968 20979 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:28.151469 16508 raft_consensus.cc:2354] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 6. Current term is 7.
I20260810 18:52:28.152544 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:27.633253 20758 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:28.154554 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:28.154599 20758 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:28.156301 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.157086 20758 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:27.062335 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:28.157591 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.158049 16632 raft_consensus.cc:2454] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:28.158612 16632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1592) took 1168 ms. Trace:
I20260810 18:52:27.698794 17787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55602 (request call id 389) took 1561 ms. Trace:
I20260810 18:52:28.159353 17787 rpcz_store.cc:276] 0810 18:52:26.137673 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:26.137868 (+ 195us) service_pool.cc:227] Handling call
0810 18:52:26.144837 (+ 6969us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:26.836988 (+692151us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:26.837031 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:26.837089 (+ 58us) raft_consensus.cc:1631] Marking committed up to 7
0810 18:52:26.837133 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:27.698104 (+860971us) spinlock_profiling.cc:243] Waited 692 ms on lock 0x7b54006b1554. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:27.698190 (+ 86us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:27.698537 (+ 347us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1591183232}
W20260810 18:52:27.707798 16443 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.644s user 0.006s sys 0.006s
W20260810 18:52:28.161243 16443 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.098s user 0.007s sys 0.007s
I20260810 18:52:28.161998 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.162878 16495 raft_consensus.cc:2379] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 8: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:27.720305 16636 raft_consensus.cc:2454] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 5.
W20260810 18:52:27.067642 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 69187.9us
I20260810 18:52:28.164799 16636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 256) took 1178 ms. Trace:
I20260810 18:52:28.165194 16636 rpcz_store.cc:276] 0810 18:52:26.986328 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:26.986479 (+ 151us) service_pool.cc:227] Handling call
0810 18:52:28.164697 (+1178218us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:28.165800 16701 leader_election.cc:302] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:28.166550 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:28.179580 16636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 262) took 1100 ms. Trace:
I20260810 18:52:27.018042 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:28.180639 16748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37716 (request call id 354) took 1169 ms. Trace:
I20260810 18:52:28.180991 16748 rpcz_store.cc:276] 0810 18:52:27.010976 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:27.012938 (+ 1962us) service_pool.cc:227] Handling call
0810 18:52:28.180411 (+1167473us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:27.115543 17724 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:28.182602 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 1.06695e+06us
W20260810 18:52:27.760262 19907 consensus_meta.cc:287] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.742s user 0.014s sys 0.006s
W20260810 18:52:27.762108 20971 log.cc:924] Time spent T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.863s user 0.000s sys 0.004s
I20260810 18:52:28.184306 19907 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:27.775576 20940 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:28.185241 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:27.780717 20891 raft_consensus.cc:482] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:27.786970 20459 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:27.792683 20907 raft_consensus.cc:482] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:27.793129 17783 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55602 (request call id 387) took 2199 ms. Trace:
I20260810 18:52:27.797773 20994 leader_election.cc:288] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:27.061980 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:28.191042 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 1.0651e+06us
I20260810 18:52:28.191825 16752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37716 (request call id 353) took 1189 ms. Trace:
I20260810 18:52:28.191536 20907 raft_consensus.cc:504] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:28.192152 16752 rpcz_store.cc:276] 0810 18:52:27.002077 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:27.010435 (+ 8358us) service_pool.cc:227] Handling call
0810 18:52:28.191638 (+1181203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:27.119621 16743 raft_consensus.cc:2454] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 4.
I20260810 18:52:27.812165 20890 leader_election.cc:288] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:28.193645 16743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37716 (request call id 350) took 1237 ms. Trace:
I20260810 18:52:28.193976 20907 leader_election.cc:288] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:27.159082 16626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1877) took 1338 ms. Trace:
I20260810 18:52:27.869939 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:52:27.008922 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 146193us
I20260810 18:52:28.194882 20891 raft_consensus.cc:504] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:28.188139 17783 rpcz_store.cc:276] 0810 18:52:25.593628 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:25.593812 (+ 184us) service_pool.cc:227] Handling call
0810 18:52:26.024949 (+431137us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:26.025173 (+ 224us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:26.025210 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:26.025263 (+ 53us) raft_consensus.cc:1631] Marking committed up to 6
0810 18:52:26.025305 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:26.025373 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:27.792654 (+1767281us) spinlock_profiling.cc:243] Waited 431 ms on lock 0x7b54005c03d0. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:27.792890 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":990629632}
W20260810 18:52:27.894483 16624 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1572) took 1507 ms (client timeout 1553 ms). Trace:
W20260810 18:52:28.196697 16624 rpcz_store.cc:269] 0810 18:52:26.386743 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:26.433172 (+ 46429us) service_pool.cc:227] Handling call
0810 18:52:26.933329 (+500157us) spinlock_profiling.cc:243] Waited 1.85 ms on lock 0x7b5800004298. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519abce 000000000034aea9 00007f1aecfc6a30 00007f1aed08c7b4 00007f1aed08c821 00007f1aed03c1b0 00007f1aed03ba94 00007f1ae6723c04 00007f1ae6723b92
0810 18:52:27.893974 (+960645us) spinlock_profiling.cc:243] Waited 23.3 ms on lock 0x7b54003c8554. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:27.894277 (+ 303us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":57864704}
I20260810 18:52:28.197319 17724 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:28.197608 20891 leader_election.cc:288] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:26.994890 19587 raft_consensus.cc:3037] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:28.197983 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:27.905839 20536 raft_consensus.cc:482] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:28.199448 20948 raft_consensus.cc:2781] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:28.199234 20536 raft_consensus.cc:504] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:28.199851 20948 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:28.200245 20948 raft_consensus.cc:3037] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:28.194876 16626 rpcz_store.cc:276] 0810 18:52:25.820531 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:25.820662 (+ 131us) service_pool.cc:227] Handling call
0810 18:52:25.839165 (+ 18503us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:25.988071 (+148906us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:25.988100 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:25.988136 (+ 36us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:25.988167 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:27.158804 (+1170637us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b540035e154. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:27.158858 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:27.159001 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":342027136}
I20260810 18:52:28.204355 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:28.204890 17724 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:28.205842 20459 raft_consensus.cc:2726] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:27.905787 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:28.186920 19907 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:28.206851 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:28.207551 16623 raft_consensus.cc:2454] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:28.143035 20927 consensus_queue.cc:237] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:28.208051 16626 raft_consensus.cc:2454] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 5.
I20260810 18:52:27.912863 16750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 1941) took 2437 ms. Trace:
I20260810 18:52:27.017187 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:28.004838 21010 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:28.210084 16750 rpcz_store.cc:276] 0810 18:52:25.474841 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:25.477853 (+ 3012us) service_pool.cc:227] Handling call
0810 18:52:25.478225 (+ 372us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:26.436788 (+958563us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:27.912384 (+1475596us) spinlock_profiling.cc:243] Waited 923 ms on lock 0x7b54003be554. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:27.912651 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2122798336}
I20260810 18:52:28.211282 16628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1593) took 1220 ms. Trace:
I20260810 18:52:28.211078 21010 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:28.211730 16628 rpcz_store.cc:276] 0810 18:52:26.990846 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:27.016943 (+ 26097us) service_pool.cc:227] Handling call
0810 18:52:28.210997 (+1194054us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:28.015393 21011 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
W20260810 18:52:28.016909 16447 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.017s user 0.007s sys 0.005s
I20260810 18:52:28.210461 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:28.052074 17770 raft_consensus.cc:2379] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 6: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:52:28.067819 20942 raft_consensus.cc:3037] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:28.077775 16633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 1876) took 2257 ms. Trace:
I20260810 18:52:28.123732 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.123939 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.126271 16742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 351) took 1154 ms. Trace:
I20260810 18:52:28.207123 20945 raft_consensus.cc:2781] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:28.127722 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:28.215621 17770 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55602 (request call id 388) took 2092 ms (client timeout 2203 ms). Trace:
I20260810 18:52:28.215796 20945 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:28.215914 20536 leader_election.cc:288] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:28.216315 20945 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Advancing to term 7
W20260810 18:52:28.216140 17770 rpcz_store.cc:269] 0810 18:52:26.123601 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:26.133752 (+ 10151us) service_pool.cc:227] Handling call
0810 18:52:28.215210 (+2081458us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b540057fdd4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:28.215467 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3229889408}
I20260810 18:52:28.216784 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.041852 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:52:28.141799 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.217939 16500 raft_consensus.cc:2454] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:28.120608 20902 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:28.218919 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.219514 20902 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:28.220216 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:27.607357 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:28.167524 20986 raft_consensus.cc:2781] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:28.222081 20902 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:28.222293 20986 raft_consensus.cc:482] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:28.222817 20986 raft_consensus.cc:3037] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:28.223125 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.167196 17726 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:52:28.167924 16701 leader_election.cc:302] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:28.143638 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.175828 20979 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:28.158924 16632 rpcz_store.cc:276] 0810 18:52:26.990582 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:27.011692 (+ 21110us) service_pool.cc:227] Handling call
0810 18:52:28.158539 (+1146847us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:27.120829 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:28.234546 20994 raft_consensus.cc:2781] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:52:28.240338 20994 raft_consensus.cc:482] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:28.240810 20994 raft_consensus.cc:3037] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:52:28.241014 16632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60066 (request call id 261) took 1252 ms. Trace:
I20260810 18:52:28.241412 16632 rpcz_store.cc:276] 0810 18:52:26.988569 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:28.240088 (+1251519us) service_pool.cc:227] Handling call
0810 18:52:28.240457 (+ 369us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:28.240709 (+ 252us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:28.240733 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:28.240767 (+ 34us) raft_consensus.cc:1631] Marking committed up to 7
0810 18:52:28.240795 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:28.240845 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:28.240939 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:28.189277 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.243520 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:28.243996 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:28.244232 16750 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
I20260810 18:52:28.245177 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.246325 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:28.234347 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 67443.7us
I20260810 18:52:28.234980 20694 raft_consensus.cc:2781] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:28.247948 20694 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:28.248396 20694 raft_consensus.cc:3037] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:28.249028 16701 leader_election.cc:302] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:28.250176 20912 raft_consensus.cc:2781] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:28.250368 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:28.250640 20912 raft_consensus.cc:482] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:28.251121 20912 raft_consensus.cc:3037] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
W20260810 18:52:28.234714 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 67962.8us
I20260810 18:52:28.252104 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:28.190621 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.253162 16747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37716 (request call id 356) took 1133 ms. Trace:
I20260810 18:52:28.253504 16747 rpcz_store.cc:276] 0810 18:52:27.119508 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:27.120502 (+ 994us) service_pool.cc:227] Handling call
0810 18:52:28.252972 (+1132470us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":36480}
I20260810 18:52:28.235553 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.254397 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:28.192363 17727 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:28.193886 16743 rpcz_store.cc:276] 0810 18:52:26.956472 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:26.956808 (+ 336us) service_pool.cc:227] Handling call
0810 18:52:28.193419 (+1236611us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b540009ef54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:28.193526 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":250233216}
I20260810 18:52:27.814496 20989 raft_consensus.cc:2726] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:28.256199 20459 raft_consensus.cc:2726] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260810 18:52:28.256239 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 64081.9us
W20260810 18:52:28.042547 16631 consensus_meta.cc:287] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 1.040s user 0.019s sys 0.032s
I20260810 18:52:28.214614 16633 rpcz_store.cc:276] 0810 18:52:25.820155 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:25.822990 (+ 2835us) service_pool.cc:227] Handling call
0810 18:52:25.884825 (+ 61835us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:26.279834 (+395009us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:26.279878 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:26.279935 (+ 57us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:26.279989 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:28.077219 (+1797230us) spinlock_profiling.cc:243] Waited 376 ms on lock 0x7b54003688d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:28.077314 (+ 95us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:28.077562 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":864373632}
I20260810 18:52:28.256975 16631 raft_consensus.cc:504] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:28.257578 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:28.258208 16633 raft_consensus.cc:3037] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:27.998665 16573 leader_election.cc:302] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:28.208112 16623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1591) took 1217 ms. Trace:
I20260810 18:52:28.259035 16631 leader_election.cc:288] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:28.183411 16636 rpcz_store.cc:276] 0810 18:52:27.079636 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:28.166164 (+1086528us) service_pool.cc:227] Handling call
0810 18:52:28.179493 (+ 13329us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 18:52:27.078617 16672 heartbeater.cc:644] Failed to heartbeat to 127.14.32.254:46351 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.14.32.254:46351 timed out after 15.000s (SENT)
I20260810 18:52:28.261493 16631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:53870 (request call id 1895) took 1285 ms. Trace:
I20260810 18:52:28.261842 16631 rpcz_store.cc:276] 0810 18:52:26.975674 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:26.977695 (+ 2021us) service_pool.cc:227] Handling call
0810 18:52:28.261319 (+1283624us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:28.222054 16621 raft_consensus.cc:2454] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:28.262177 16636 raft_consensus.cc:1226] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:28.262678 16621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1588) took 1273 ms. Trace:
I20260810 18:52:28.168529 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.263267 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:28.263635 16636 raft_consensus.cc:1226] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: []
W20260810 18:52:28.214011 16447 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.214s user 0.008s sys 0.005s
I20260810 18:52:28.263859 16495 raft_consensus.cc:2379] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 6: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:28.212809 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:28.185993 20940 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:28.195611 16622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1589) took 1205 ms. Trace:
I20260810 18:52:28.265269 16628 raft_consensus.cc:2379] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 7: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:28.233542 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:28.265416 16622 rpcz_store.cc:276] 0810 18:52:26.989673 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:27.869301 (+879628us) service_pool.cc:227] Handling call
0810 18:52:28.195504 (+326203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:28.213613 21011 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:28.259943 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 261719us
I20260810 18:52:28.263017 16621 rpcz_store.cc:276] 0810 18:52:26.989421 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:27.606524 (+617103us) service_pool.cc:227] Handling call
0810 18:52:28.262625 (+656101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:28.268208 16621 raft_consensus.cc:1226] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:28.268977 20940 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:28.269018 16573 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:28.252149 17726 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:52:28.260603 20989 raft_consensus.cc:2726] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:28.271687 20774 raft_consensus.cc:2781] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:28.271849 16624 raft_consensus.cc:3037] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:28.272123 20774 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:28.272549 20774 raft_consensus.cc:3037] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:28.274188 16622 raft_consensus.cc:1226] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:28.215538 16742 rpcz_store.cc:276] 0810 18:52:26.971629 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:26.971797 (+ 168us) service_pool.cc:227] Handling call
0810 18:52:26.972176 (+ 379us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:28.025863 (+1053687us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:28.025922 (+ 59us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:28.025987 (+ 65us) raft_consensus.cc:1631] Marking committed up to 7
0810 18:52:28.026049 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:28.125736 (+ 99687us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b540034e9d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:28.125779 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:28.126011 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2422035200}
I20260810 18:52:28.217962 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.221489 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.190137 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.264513 21010 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:28.250929 16701 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:28.260754 16623 rpcz_store.cc:276] 0810 18:52:26.990314 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:27.902641 (+912327us) service_pool.cc:227] Handling call
0810 18:52:28.208054 (+305413us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:28.276849 16750 raft_consensus.cc:2454] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
I20260810 18:52:28.270722 16573 leader_election.cc:302] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:28.239701 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.268862 21011 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:28.287309 16632 raft_consensus.cc:1226] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:28.299723 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.301308 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.302820 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:28.303220 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 52416.4us
I20260810 18:52:28.305519 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.306861 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.308171 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.309185 20940 raft_consensus.cc:2781] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:28.309584 20940 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:28.309866 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.309898 20940 raft_consensus.cc:3037] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:28.323246 16496 raft_consensus.cc:2454] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:28.312000 16573 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:28.326102 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.312304 16701 leader_election.cc:302] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:28.317610 16746 raft_consensus.cc:2396] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260810 18:52:28.312871 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.328629 16495 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
I20260810 18:52:28.319025 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.315639 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.313151 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.333806 16573 leader_election.cc:302] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:28.314240 20902 consensus_queue.cc:1046] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
W20260810 18:52:28.335665 16746 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37716 (request call id 352) took 1350 ms (client timeout 1774 ms). Trace:
I20260810 18:52:28.336249 20902 raft_consensus.cc:3032] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Stepping down as leader of term 3
W20260810 18:52:28.336128 16746 rpcz_store.cc:269] 0810 18:52:26.985722 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:27.015755 (+ 30033us) service_pool.cc:227] Handling call
0810 18:52:28.335457 (+1319702us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b540009f954. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:28.335608 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2991159680}
I20260810 18:52:28.336853 20902 raft_consensus.cc:729] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:28.337384 20989 raft_consensus.cc:2726] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:28.337903 20902 consensus_queue.cc:260] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:28.316497 20898 consensus_queue.cc:1046] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20260810 18:52:28.332598 16624 raft_consensus.cc:2454] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 7.
W20260810 18:52:28.345793 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 67279.9us
I20260810 18:52:28.345916 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:28.318693 20694 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:28.351877 16573 leader_election.cc:302] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:28.353147 16750 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:28.356073 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:28.358574 20694 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:28.359766 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.359579 20945 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:28.339455 21011 raft_consensus.cc:2726] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:28.361819 16633 raft_consensus.cc:2454] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
I20260810 18:52:28.327430 21010 raft_consensus.cc:2726] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:28.364223 20942 raft_consensus.cc:2781] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:28.364467 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.364660 20942 raft_consensus.cc:482] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:28.365178 20942 raft_consensus.cc:3037] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:28.365669 20979 raft_consensus.cc:2741] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20260810 18:52:28.368053 16573 leader_election.cc:302] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:28.367918 20940 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:28.369339 16573 leader_election.cc:302] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:28.310863 20986 raft_consensus.cc:504] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:28.370575 20989 raft_consensus.cc:2726] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:52:28.371260 20536 raft_consensus.cc:2726] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:28.370105 20891 raft_consensus.cc:2726] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:28.371740 20986 leader_election.cc:288] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:28.372381 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.372712 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.372965 20942 raft_consensus.cc:504] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:28.373453 16505 raft_consensus.cc:2454] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 7.
I20260810 18:52:28.370625 19907 raft_consensus.cc:3032] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:28.374182 19907 raft_consensus.cc:729] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:28.374850 20942 leader_election.cc:288] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:28.375156 19907 consensus_queue.cc:260] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:28.376456 19907 raft_consensus.cc:3037] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:28.376807 20994 raft_consensus.cc:504] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:28.378764 20994 leader_election.cc:288] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:28.379743 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:28.380661 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.380722 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:28.370052 20940 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:28.384320 16573 leader_election.cc:302] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:28.384125 20912 raft_consensus.cc:504] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:28.347656 20774 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:28.392452 20945 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:28.400565 16701 leader_election.cc:302] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:28.415680 16500 raft_consensus.cc:2454] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
I20260810 18:52:28.402252 17727 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:28.394923 16746 raft_consensus.cc:1226] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:28.418244 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:28.424031 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:28.424515 16750 raft_consensus.cc:3037] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:28.394593 19587 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:28.423213 21010 raft_consensus.cc:2726] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:52:28.425637 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:28.425843 16701 leader_election.cc:302] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:28.426368 17770 raft_consensus.cc:2454] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:28.426725 20994 raft_consensus.cc:2781] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:28.427018 17783 raft_consensus.cc:1226] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:28.427208 20994 raft_consensus.cc:482] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:28.427767 20994 raft_consensus.cc:3037] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:28.410072 20902 raft_consensus.cc:3037] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:28.425361 20948 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:28.432418 16345 catalog_manager.cc:5495] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 4 to 5, leader changed from 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 5 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:52:28.431207 16743 raft_consensus.cc:2379] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 6: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
W20260810 18:52:28.443163 16450 connection.cc:232] RPC call timeout handler was delayed by -1.1670722218696028s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17720000000000002s for a response
I20260810 18:52:28.443858 20948 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:28.442142 20940 raft_consensus.cc:2781] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260810 18:52:28.453794 16450 connection.cc:232] RPC call timeout handler was delayed by -1.1607722218614072s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1779s for a response
I20260810 18:52:28.447844 20774 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:28.427318 16701 leader_election.cc:302] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:28.452090 19587 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:28.453971 20940 raft_consensus.cc:482] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:28.446593 20994 raft_consensus.cc:504] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:28.460901 20940 raft_consensus.cc:3037] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:28.462271 20994 leader_election.cc:288] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:28.458598 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:28.462951 17783 raft_consensus.cc:2454] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
W20260810 18:52:28.459189 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 148248us
I20260810 18:52:28.459501 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:28.460352 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:28.464030 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:28.462077 20952 consensus_queue.cc:1046] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:52:28.465015 17778 raft_consensus.cc:2379] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 7: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:52:28.465582 20952 raft_consensus.cc:3032] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:28.465965 20952 raft_consensus.cc:729] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:28.466964 20952 consensus_queue.cc:260] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:28.467537 20994 raft_consensus.cc:2781] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:28.468027 20994 raft_consensus.cc:482] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:28.468429 20994 raft_consensus.cc:3037] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:28.468402 16701 leader_election.cc:302] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:28.468333 20952 raft_consensus.cc:3037] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
W20260810 18:52:28.454504 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 70373.7us
W20260810 18:52:28.471669 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 69527.5us
I20260810 18:52:28.473750 20948 raft_consensus.cc:2781] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:28.474224 20948 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:28.474627 20948 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:28.478088 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:28.481536 17787 raft_consensus.cc:1226] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 3. Current term is 4. Ops: []
W20260810 18:52:28.482542 16387 debug-util.cc:398] Leaking SignalData structure 0x7b080038ac00 after lost signal to thread 14470
W20260810 18:52:28.483816 16387 debug-util.cc:398] Leaking SignalData structure 0x7b080031b400 after lost signal to thread 16391
W20260810 18:52:28.484179 16387 debug-util.cc:398] Leaking SignalData structure 0x7b080037e1c0 after lost signal to thread 16415
W20260810 18:52:28.485476 16387 debug-util.cc:398] Leaking SignalData structure 0x7b0800399440 after lost signal to thread 16545
W20260810 18:52:28.485734 16387 debug-util.cc:398] Leaking SignalData structure 0x7b08003a4880 after lost signal to thread 16546
I20260810 18:52:28.489459 20940 raft_consensus.cc:504] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:28.490794 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:28.491259 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:28.493248 17770 raft_consensus.cc:1226] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:28.495342 17783 raft_consensus.cc:1226] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:28.495607 16750 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
I20260810 18:52:28.496913 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:28.498301 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:28.471778 16345 catalog_manager.cc:5495] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 5 to 6, leader changed from 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 6 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:52:28.503795 20898 consensus_queue.cc:1046] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260810 18:52:28.486886 16387 debug-util.cc:398] Leaking SignalData structure 0x7b080037b340 after lost signal to thread 16671
I20260810 18:52:28.505795 20854 raft_consensus.cc:3032] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Stepping down as leader of term 5
W20260810 18:52:28.505972 16387 debug-util.cc:398] Leaking SignalData structure 0x7b08002e55c0 after lost signal to thread 16786
I20260810 18:52:28.506238 20854 raft_consensus.cc:729] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:28.506079 20940 leader_election.cc:288] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:28.506696 16387 debug-util.cc:398] Leaking SignalData structure 0x7b08002e5560 after lost signal to thread 16787
I20260810 18:52:28.507373 20942 raft_consensus.cc:2781] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:28.507855 20942 raft_consensus.cc:482] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:28.508296 20942 raft_consensus.cc:3037] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:28.508138 16701 leader_election.cc:302] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:28.509161 20986 raft_consensus.cc:2781] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:28.509538 20986 raft_consensus.cc:482] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:28.509867 20986 raft_consensus.cc:3037] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:28.510201 20912 leader_election.cc:288] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:28.515527 16701 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
W20260810 18:52:28.516032 16387 debug-util.cc:398] Leaking SignalData structure 0x7b08003822e0 after lost signal to thread 17827
W20260810 18:52:28.516335 16387 debug-util.cc:398] Leaking SignalData structure 0x7b080037eca0 after lost signal to thread 17829
I20260810 18:52:28.519272 16573 leader_election.cc:302] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:28.520260 20927 raft_consensus.cc:2726] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260810 18:52:28.520491 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 47103.3us
I20260810 18:52:28.519608 20942 raft_consensus.cc:504] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:28.521302 16448 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:28.507505 20854 consensus_queue.cc:260] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:28.542889 20854 raft_consensus.cc:3037] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:28.543440 20898 raft_consensus.cc:2781] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:28.592916 20898 raft_consensus.cc:686] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 6 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:28.593824 20898 consensus_queue.cc:237] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:28.495261 20948 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:28.493319 17727 leader_election.cc:302] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:28.546821 20912 raft_consensus.cc:2781] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:28.599887 20912 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:28.600337 20912 raft_consensus.cc:3037] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:28.600772 20948 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:28.603222 17778 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
I20260810 18:52:28.548147 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:52:28.550865 16343 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37026 (request call id 226) took 16827 ms (client timeout 15000 ms). Trace:
W20260810 18:52:28.605898 16343 rpcz_store.cc:269] 0810 18:52:11.723561 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:11.755211 (+ 31650us) service_pool.cc:227] Handling call
0810 18:52:28.550576 (+16795365us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 18:52:23.369572 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 18:52:23.779009 (+409437us) write_op.cc:432] Acquiring schema lock in shared mode
0810 18:52:23.783814 (+ 4805us) write_op.cc:435] Acquired schema lock
0810 18:52:23.783842 (+ 28us) tablet.cc:660] Decoding operations
0810 18:52:24.237751 (+453909us) write_op.cc:620] Acquiring the partition lock for write op
0810 18:52:24.237841 (+ 90us) write_op.cc:641] Partition lock acquired for write op
0810 18:52:24.237895 (+ 54us) tablet.cc:683] Acquiring locks for 17 operations
0810 18:52:24.299334 (+ 61439us) tablet.cc:699] Row locks acquired
0810 18:52:24.299358 (+ 24us) write_op.cc:260] PREPARE: finished
0810 18:52:24.299504 (+ 146us) write_op.cc:270] Start()
0810 18:52:24.299646 (+ 142us) write_op.cc:276] Timestamp: P: 1786387944299463 usec, L: 0
0810 18:52:24.299669 (+ 23us) op_driver.cc:331] REPLICATION: starting
0810 18:52:24.300323 (+ 654us) log.cc:842] Serialized 6793 byte log entry
0810 18:52:26.063268 (+1762945us) op_driver.cc:444] REPLICATION: finished
0810 18:52:26.936512 (+873244us) write_op.cc:301] APPLY: starting
0810 18:52:26.936608 (+ 96us) tablet.cc:1365] starting BulkCheckPresence
0810 18:52:26.936955 (+ 347us) tablet.cc:1368] finished BulkCheckPresence
0810 18:52:26.937107 (+ 152us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 18:52:27.072111 (+135004us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 18:52:27.072544 (+ 433us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=17
0810 18:52:27.072618 (+ 74us) write_op.cc:312] APPLY: finished
0810 18:52:27.073126 (+ 508us) log.cc:842] Serialized 119 byte log entry
0810 18:52:27.166424 (+ 93298us) write_op.cc:489] Releasing partition, row and schema locks
0810 18:52:27.179643 (+ 13219us) write_op.cc:454] Released schema lock
0810 18:52:27.266615 (+ 86972us) write_op.cc:341] FINISH: Updating metrics
0810 18:52:27.279464 (+ 12849us) write_op.cc:489] Releasing partition, row and schema locks
0810 18:52:27.279509 (+ 45us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1799709,"reported_tablets":38,"child_traces":[["op",{"apply.queue_time_us":859796,"apply.run_cpu_time_us":16069,"apply.run_wall_time_us":343548,"num_ops":17,"prepare.queue_time_us":195995,"prepare.run_cpu_time_us":84850,"prepare.run_wall_time_us":1288880,"raft.queue_time_us":576428,"raft.run_cpu_time_us":132773,"raft.run_wall_time_us":949831,"replication_time_us":1757161,"thread_start_us":1810714,"threads_started":5,"wal-append.queue_time_us":1067986,"wal-append.run_cpu_time_us":25638,"wal-append.run_wall_time_us":2825791}]]}
I20260810 18:52:28.547796 20942 leader_election.cc:288] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:28.578225 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:28.580866 20986 raft_consensus.cc:504] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:28.558718 16695 connection.cc:232] RPC call timeout handler was delayed by -1.5275220312178135s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21440000000000003s for a response
I20260810 18:52:28.582132 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
W20260810 18:52:28.585688 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 162379us
I20260810 18:52:28.614235 17780 raft_consensus.cc:3037] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:28.588114 17787 raft_consensus.cc:1226] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:28.589736 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:52:28.620201 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 64759.6us
I20260810 18:52:28.590997 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:52:28.621089 16695 connection.cc:232] RPC call timeout handler was delayed by -1.504122031154111s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.217s for a response
I20260810 18:52:28.592200 17770 raft_consensus.cc:1226] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:28.599534 20994 raft_consensus.cc:504] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:28.580230 16345 catalog_manager.cc:5495] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:52:28.614257 17783 raft_consensus.cc:2454] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
W20260810 18:52:28.621510 16695 connection.cc:232] RPC call timeout handler was delayed by -1.44202203117311s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2239s for a response
W20260810 18:52:28.620774 16575 leader_election.cc:334] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Timed out: RequestConsensusVote RPC to 127.14.32.195:40171 timed out after 1.557s (SENT)
W20260810 18:52:28.625033 16695 connection.cc:232] RPC call timeout handler was delayed by -1.2630172139033675s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15300000000000002s for a response
W20260810 18:52:28.625429 16695 connection.cc:232] RPC call timeout handler was delayed by -1.2468172139488161s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15480000000000002s for a response
I20260810 18:52:28.625254 16575 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:28.625752 16695 connection.cc:232] RPC call timeout handler was delayed by -1.2459172140806913s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1549s for a response
W20260810 18:52:28.626101 16695 connection.cc:232] RPC call timeout handler was delayed by -1.2423172139097005s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1553s for a response
W20260810 18:52:28.630605 16695 connection.cc:232] RPC call timeout handler was delayed by -1.1325172139331698s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1675s for a response
W20260810 18:52:28.631165 16695 connection.cc:232] RPC call timeout handler was delayed by -1.0659172139130533s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17490000000000003s for a response
W20260810 18:52:28.631671 16695 connection.cc:232] RPC call timeout handler was delayed by -1.0479172139894217s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1769s for a response
I20260810 18:52:28.632931 21017 raft_consensus.cc:2781] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:28.633080 20940 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:28.633441 21017 raft_consensus.cc:482] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:28.633868 21017 raft_consensus.cc:3037] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:28.633793 20940 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:28.636207 20940 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:28.637319 20797 raft_consensus.cc:482] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:28.637913 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 144695us
I20260810 18:52:28.638065 20797 raft_consensus.cc:504] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:28.639266 17727 leader_election.cc:302] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:28.631800 16573 leader_election.cc:302] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:28.628027 20942 raft_consensus.cc:482] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:28.644829 16345 catalog_manager.cc:5495] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 2 to 5, leader changed from d304ac47fdf846de81337057a4ef7bea (127.14.32.193) to 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196). New cstate: current_term: 5 leader_uuid: "831925dd8b644ebab7eb0d672a94eddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:52:28.637388 20894 raft_consensus.cc:2726] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260810 18:52:28.619228 20912 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:28.645704 17724 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:28.654949 17726 leader_election.cc:302] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:52:28.657582 20797 leader_election.cc:288] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:28.658118 20797 raft_consensus.cc:482] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:28.658476 20758 raft_consensus.cc:482] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:28.658557 20797 raft_consensus.cc:504] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:28.659529 20758 raft_consensus.cc:504] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:28.662654 20758 leader_election.cc:288] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:28.664183 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 86983.7us
I20260810 18:52:28.649737 20927 raft_consensus.cc:2726] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:28.669008 16450 leader_election.cc:302] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:28.650272 20948 raft_consensus.cc:2726] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260810 18:52:28.656260 20894 raft_consensus.cc:482] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:28.675767 20986 leader_election.cc:288] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:28.672629 20459 raft_consensus.cc:2781] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader election won for term 7
I20260810 18:52:28.678601 20994 leader_election.cc:288] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:28.668308 20774 raft_consensus.cc:2726] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:28.678346 20894 raft_consensus.cc:504] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:28.680685 20940 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:28.681578 20894 leader_election.cc:288] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:28.673027 20942 raft_consensus.cc:504] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:28.687543 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:28.688313 20942 leader_election.cc:288] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:28.689339 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 43690.4us
I20260810 18:52:28.689910 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:28.690653 17724 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:28.681222 20940 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:28.691987 17724 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:28.693022 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 74063.8us
I20260810 18:52:28.692667 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:28.691187 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:28.668104 16345 catalog_manager.cc:5495] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 2 to 7, leader changed from <none> to 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196). New cstate: current_term: 7 leader_uuid: "831925dd8b644ebab7eb0d672a94eddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:52:28.707571 16747 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:28.719043 16447 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:28.691795 20912 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:28.718232 16345 catalog_manager.cc:5495] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 5 to 6, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 6 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:52:28.692253 20945 raft_consensus.cc:2781] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:28.692672 20797 leader_election.cc:288] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:28.705914 20459 raft_consensus.cc:686] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [term 7 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:28.733013 20694 raft_consensus.cc:2726] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260810 18:52:28.742296 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 87865.4us
I20260810 18:52:28.742533 20459 consensus_queue.cc:237] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:28.741354 20945 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:28.744297 16450 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:52:28.740331 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 48433.1us
I20260810 18:52:28.749112 19907 consensus_queue.cc:1046] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260810 18:52:28.749801 17724 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:28.750065 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:28.750478 19907 raft_consensus.cc:2726] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:28.751199 19907 raft_consensus.cc:3032] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:28.751564 19907 raft_consensus.cc:729] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:28.751672 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:28.752526 19907 consensus_queue.cc:260] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:28.753106 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:28.753728 16622 raft_consensus.cc:3037] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:28.754159 19907 raft_consensus.cc:3037] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:28.754094 21017 raft_consensus.cc:504] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:28.755302 20898 consensus_queue.cc:1046] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260810 18:52:28.755555 19587 consensus_queue.cc:1046] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 18:52:28.757768 20898 raft_consensus.cc:3032] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:28.758270 20898 raft_consensus.cc:729] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:28.760313 20948 raft_consensus.cc:2781] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:28.760823 20948 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:28.761309 20948 raft_consensus.cc:3037] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:28.747574 16345 catalog_manager.cc:5495] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 3 to 4, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 4 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:52:28.773692 21017 leader_election.cc:288] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:28.737411 20902 raft_consensus.cc:2781] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:28.775368 20902 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:28.744756 20858 raft_consensus.cc:2726] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:28.764820 16345 catalog_manager.cc:5495] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
W20260810 18:52:28.791998 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 150239us
I20260810 18:52:28.799245 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
W20260810 18:52:28.799666 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 52805.1us
I20260810 18:52:28.799921 16742 raft_consensus.cc:3037] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:28.801492 16747 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
I20260810 18:52:28.809758 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:28.802676 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:28.802337 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:28.744302 20945 raft_consensus.cc:3037] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:28.811278 16632 raft_consensus.cc:2379] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 6: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:28.812894 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:28.814468 20940 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:28.819294 20898 consensus_queue.cc:260] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:28.820346 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:28.820772 16573 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:28.821158 21028 raft_consensus.cc:482] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:28.822042 20797 raft_consensus.cc:482] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:28.823608 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:28.841526 16752 raft_consensus.cc:2379] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 7: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:28.841156 21028 raft_consensus.cc:504] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:28.824347 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:28.842396 16743 raft_consensus.cc:2454] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 7.
I20260810 18:52:28.824870 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:28.838899 16742 raft_consensus.cc:2454] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 5.
I20260810 18:52:28.826267 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:28.826841 17780 raft_consensus.cc:2454] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
I20260810 18:52:28.827795 21004 raft_consensus.cc:3032] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:28.828475 20994 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:28.786275 20902 raft_consensus.cc:3037] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:28.830839 16575 leader_election.cc:302] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:28.831594 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:28.832341 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:28.834031 16447 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
W20260810 18:52:28.835321 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 49096.2us
I20260810 18:52:28.831367 16345 catalog_manager.cc:5495] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 5 to 7, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 7 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:52:28.838460 16695 leader_election.cc:302] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:28.839831 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:28.840495 20797 raft_consensus.cc:504] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:28.862571 20797 leader_election.cc:288] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:28.825052 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:28.844916 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:28.863348 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:28.863847 16746 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 8.
I20260810 18:52:28.864941 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:28.865552 16746 raft_consensus.cc:3037] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:28.848619 17727 leader_election.cc:302] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:28.848820 16749 raft_consensus.cc:2379] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 4: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:28.846688 21004 raft_consensus.cc:729] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:28.866482 21017 raft_consensus.cc:2781] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:28.867206 21004 consensus_queue.cc:260] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:28.867975 16622 raft_consensus.cc:2454] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:28.868671 21004 raft_consensus.cc:3037] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:28.868896 16575 leader_election.cc:302] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:28.869347 21017 raft_consensus.cc:686] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [term 5 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:28.869728 20758 raft_consensus.cc:2726] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:28.850342 16500 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0"
dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
mode: GRACEFUL
new_leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
from {username='slave'} at 127.0.0.1:51572
I20260810 18:52:28.870386 21017 consensus_queue.cc:237] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:28.871368 16500 raft_consensus.cc:595] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Received request to transfer leadership to TS 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:28.871747 16500 raft_consensus.cc:603] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260810 18:52:28.871991 16575 leader_election.cc:302] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:28.850068 20994 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:28.851583 21028 leader_election.cc:288] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:28.846457 20894 raft_consensus.cc:2781] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:28.857136 20898 raft_consensus.cc:3037] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
W20260810 18:52:28.877664 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 43683.9us
I20260810 18:52:28.878401 20894 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:28.882417 20927 raft_consensus.cc:2726] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:28.870379 20758 raft_consensus.cc:2781] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:28.883458 16345 catalog_manager.cc:5495] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 3 to 4, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 4 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
W20260810 18:52:28.885974 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 48179.1us
I20260810 18:52:28.886888 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:28.888226 16748 raft_consensus.cc:2379] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 4: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:28.888528 16447 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:28.884361 21028 raft_consensus.cc:2726] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:28.887930 20758 raft_consensus.cc:482] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:28.890434 20758 raft_consensus.cc:3037] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:28.890218 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:28.891402 16749 raft_consensus.cc:2379] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 3: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:28.893754 20994 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:28.884722 20894 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:28.896835 16695 leader_election.cc:302] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:28.900108 16746 raft_consensus.cc:2454] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
I20260810 18:52:28.901103 16500 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8cbe74806d8349e690b392275de70292"
dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
mode: GRACEFUL
new_leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
from {username='slave'} at 127.0.0.1:51572
I20260810 18:52:28.901465 16575 leader_election.cc:302] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:28.901788 16500 raft_consensus.cc:595] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Received request to transfer leadership to TS 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:28.902583 20927 raft_consensus.cc:2726] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:28.903715 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:28.902534 20758 raft_consensus.cc:504] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:28.905032 21038 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:28.905419 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:28.905598 21038 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:28.906580 20758 leader_election.cc:288] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:28.909189 16500 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5"
dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
mode: GRACEFUL
new_leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
from {username='slave'} at 127.0.0.1:51572
I20260810 18:52:28.887183 21036 raft_consensus.cc:2726] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260810 18:52:28.910686 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:28.909334 21038 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:28.912467 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:28.913265 17771 raft_consensus.cc:2379] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 6: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:52:28.915392 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:28.917047 20948 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:28.917047 20797 raft_consensus.cc:2726] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:28.909827 16500 raft_consensus.cc:595] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 2 LEADER]: Received request to transfer leadership to TS 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:28.921819 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:28.923715 16695 leader_election.cc:398] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Vote denied by peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791) with higher term. Message: Invalid argument: T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 6. Current term is 7.
I20260810 18:52:28.923849 16500 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d"
dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
mode: GRACEFUL
new_leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
from {username='slave'} at 127.0.0.1:51572
I20260810 18:52:28.924062 16695 leader_election.cc:401] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260810 18:52:28.924886 16500 raft_consensus.cc:595] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Received request to transfer leadership to TS 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:28.925109 20948 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:28.925820 16500 raft_consensus.cc:603] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260810 18:52:28.925427 20797 raft_consensus.cc:2726] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791) with higher term. Message: Invalid argument: T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 6. Current term is 7.
I20260810 18:52:28.927488 16695 leader_election.cc:302] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:28.930262 16447 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:28.929912 16345 catalog_manager.cc:5495] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 2 to 5, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196). New cstate: current_term: 5 leader_uuid: "831925dd8b644ebab7eb0d672a94eddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:52:28.931965 21038 raft_consensus.cc:2781] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:28.932369 21038 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:28.932729 21038 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:28.933777 16749 raft_consensus.cc:1261] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260810 18:52:28.934782 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:28.935469 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:28.935935 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:28.936709 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:28.938019 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:28.938504 16345 catalog_manager.cc:5495] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:52:28.939776 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:28.940058 19907 consensus_queue.cc:1046] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:52:28.939903 16345 catalog_manager.cc:5495] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:52:28.941205 19907 raft_consensus.cc:979] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea: : Instructing follower 980f343dd1fd4e049143f8b9d9f2f1be to start an election
I20260810 18:52:28.941712 19907 raft_consensus.cc:3032] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:52:28.942101 19907 raft_consensus.cc:729] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:28.941531 16345 catalog_manager.cc:5495] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:52:28.943218 20854 raft_consensus.cc:1067] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Signalling peer 980f343dd1fd4e049143f8b9d9f2f1be to start an election
I20260810 18:52:28.943941 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:28.943445 16345 catalog_manager.cc:5495] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 3 to 4, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 4 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:52:28.944514 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:28.945175 16752 raft_consensus.cc:2454] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
I20260810 18:52:28.945497 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:28.946799 16742 raft_consensus.cc:2454] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:52:28.945175 16345 catalog_manager.cc:5495] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:52:28.944519 16746 raft_consensus.cc:3037] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:28.948192 16345 catalog_manager.cc:5495] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:52:28.948910 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:28.949465 16624 raft_consensus.cc:3037] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 5
I20260810 18:52:28.949671 16345 catalog_manager.cc:5495] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:52:28.950815 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:28.940402 16749 raft_consensus.cc:3037] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:28.951483 16345 catalog_manager.cc:5495] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:52:28.953759 16345 catalog_manager.cc:5495] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:52:28.954864 20945 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:28.942929 19907 consensus_queue.cc:260] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:28.955133 16345 catalog_manager.cc:5495] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } }
I20260810 18:52:28.955981 17780 raft_consensus.cc:1226] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 7. Current term is 8. Ops: []
I20260810 18:52:28.956225 16624 raft_consensus.cc:2454] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 5.
I20260810 18:52:28.956578 19907 raft_consensus.cc:3037] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:28.939714 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:28.957103 16345 catalog_manager.cc:5495] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 5 to 6, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 6 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:52:28.958727 16345 catalog_manager.cc:5495] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 2 to 3, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 3 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:52:28.960577 20945 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:28.960397 16345 catalog_manager.cc:5495] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 3 to 5, leader changed from d304ac47fdf846de81337057a4ef7bea (127.14.32.193) to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 5 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:52:28.961906 16695 leader_election.cc:302] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:28.963867 16695 leader_election.cc:302] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:28.962764 20912 raft_consensus.cc:2726] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:52:28.964680 20912 raft_consensus.cc:2726] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260810 18:52:28.964666 20994 raft_consensus.cc:2726] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:28.962147 16345 catalog_manager.cc:5495] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:52:28.967221 16695 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:28.967979 20994 raft_consensus.cc:2726] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:28.968734 16695 leader_election.cc:302] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:28.969728 21028 raft_consensus.cc:2726] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:28.946547 20902 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:28.966879 16345 catalog_manager.cc:5495] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 1 to 3, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 3 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:52:28.971112 16345 catalog_manager.cc:5495] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 1 to 5, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 5 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:52:28.977698 16746 raft_consensus.cc:2454] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:52:28.979470 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:28.983546 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 53350.8us
I20260810 18:52:28.985630 16447 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:28.984259 21035 raft_consensus.cc:2781] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:28.987107 21035 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:28.987238 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:28.987560 21035 raft_consensus.cc:3037] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:28.988020 16748 raft_consensus.cc:2379] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 7: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:28.971760 20902 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:28.992190 20894 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:28.996435 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:28.996752 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:28.997400 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:28.997704 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:28.999028 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:29.000424 20894 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:29.001356 16623 raft_consensus.cc:2354] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for earlier term 6. Current term is 7.
I20260810 18:52:28.929661 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:29.004086 21035 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:28.986250 20758 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:29.005053 20758 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:29.005550 16500 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd"
dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
mode: GRACEFUL
new_leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
from {username='slave'} at 127.0.0.1:51572
I20260810 18:52:29.006166 16500 raft_consensus.cc:595] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Received request to transfer leadership to TS 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:29.007387 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260810 18:52:29.009224 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:52:29.009932 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:52:29.010422 21038 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:29.011351 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:29.011803 16748 raft_consensus.cc:1261] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260810 18:52:28.990702 16450 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:29.013015 17726 leader_election.cc:398] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 election: Vote denied by peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827) with higher term. Message: Invalid argument: T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for earlier term 6. Current term is 7.
I20260810 18:52:29.013314 17726 leader_election.cc:401] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260810 18:52:29.013811 21036 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:29.015479 20902 raft_consensus.cc:979] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea: : Instructing follower 980f343dd1fd4e049143f8b9d9f2f1be to start an election
I20260810 18:52:29.015403 16450 leader_election.cc:302] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:29.016033 20902 raft_consensus.cc:1067] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Signalling peer 980f343dd1fd4e049143f8b9d9f2f1be to start an election
I20260810 18:52:29.016487 20952 raft_consensus.cc:2726] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:29.016939 16450 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:29.017594 21017 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:29.017814 16450 leader_election.cc:302] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:29.018196 20898 raft_consensus.cc:2781] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:29.018729 20898 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:29.011965 16624 raft_consensus.cc:2454] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:29.023232 20898 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:29.023921 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:29.024533 16623 raft_consensus.cc:3037] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:29.026496 21017 raft_consensus.cc:2726] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827) with higher term. Message: Invalid argument: T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for earlier term 6. Current term is 7.
I20260810 18:52:29.028980 21038 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:29.029611 16749 raft_consensus.cc:2454] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
I20260810 18:52:28.969950 16695 leader_election.cc:302] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:29.033396 20797 raft_consensus.cc:2726] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260810 18:52:29.034499 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 64613.5us
I20260810 18:52:29.035816 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:29.038261 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:29.037174 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:29.039330 16695 leader_election.cc:302] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:29.039497 16633 raft_consensus.cc:3037] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:29.006722 16345 catalog_manager.cc:5495] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 3 to 4, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 4 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:52:29.038461 16624 raft_consensus.cc:2379] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 5: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:52:29.041998 16345 catalog_manager.cc:5495] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 6 to 7, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to d304ac47fdf846de81337057a4ef7bea (127.14.32.193). New cstate: current_term: 7 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } }
I20260810 18:52:29.023118 20758 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:29.043624 16345 catalog_manager.cc:5495] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d304ac47fdf846de81337057a4ef7bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:52:29.046118 20797 raft_consensus.cc:2781] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:29.046761 20797 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:29.047255 20797 raft_consensus.cc:3037] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:29.048580 21035 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:29.049659 16573 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:29.050590 20927 raft_consensus.cc:3037] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:29.050727 20898 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:29.052892 20898 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:29.056528 20797 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:29.065599 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:29.061422 20927 raft_consensus.cc:2726] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:29.062958 16624 raft_consensus.cc:1226] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:29.064116 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:29.065272 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:29.067623 16622 raft_consensus.cc:2379] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 6: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:29.065464 16695 leader_election.cc:302] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:29.067714 20797 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:29.069984 20797 raft_consensus.cc:2726] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:29.076488 21036 raft_consensus.cc:2726] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:29.079346 21004 raft_consensus.cc:2726] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260810 18:52:29.082010 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 64262.2us
I20260810 18:52:29.082801 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:29.083510 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:29.084923 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:29.085701 17771 raft_consensus.cc:2454] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 7.
I20260810 18:52:29.087476 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:29.090237 17770 raft_consensus.cc:1226] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:29.091390 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:29.092108 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:29.093195 16622 raft_consensus.cc:2379] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 7: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:29.093840 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:29.095503 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:29.099186 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:29.100333 16575 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:29.100385 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:29.101461 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:29.102267 16624 raft_consensus.cc:2379] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 6: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:29.115990 16633 raft_consensus.cc:2454] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:29.119287 20758 consensus_queue.cc:1046] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260810 18:52:29.120807 20927 raft_consensus.cc:3032] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 7 LEADER]: Stepping down as leader of term 7
I20260810 18:52:29.121214 20927 raft_consensus.cc:729] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:29.122128 20927 consensus_queue.cc:260] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:29.124161 20927 raft_consensus.cc:3037] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Advancing to term 8
W20260810 18:52:29.131336 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 74368.3us
I20260810 18:52:29.134145 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:29.143513 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:29.150072 19907 raft_consensus.cc:2781] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election won for term 6
W20260810 18:52:29.150270 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 164713us
I20260810 18:52:29.150676 19907 raft_consensus.cc:686] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 6 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:29.151592 19907 consensus_queue.cc:237] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:29.152246 16450 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:29.153805 20952 raft_consensus.cc:2781] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election won for term 7
I20260810 18:52:29.154325 20952 raft_consensus.cc:686] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:29.155211 20952 consensus_queue.cc:237] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:29.169512 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:29.177561 16500 raft_consensus.cc:2454] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
I20260810 18:52:29.179638 16505 raft_consensus.cc:1226] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 7. Current term is 8. Ops: []
I20260810 18:52:29.181245 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:29.189472 16505 raft_consensus.cc:2454] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
I20260810 18:52:29.190599 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.191730 16500 raft_consensus.cc:2454] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
I20260810 18:52:29.194567 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:29.200798 16623 raft_consensus.cc:2454] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
W20260810 18:52:29.202072 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 49890.6us
I20260810 18:52:29.203194 19907 consensus_queue.cc:1046] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:52:29.204948 21036 raft_consensus.cc:3032] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:52:29.205303 21036 raft_consensus.cc:729] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:29.205770 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:29.206112 21036 consensus_queue.cc:260] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:29.206864 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:29.208235 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:29.209400 21036 raft_consensus.cc:3037] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:29.215830 16508 raft_consensus.cc:2454] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
I20260810 18:52:29.219362 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:29.220233 16508 raft_consensus.cc:2354] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for earlier term 6. Current term is 7.
W20260810 18:52:29.224016 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 123779us
I20260810 18:52:29.224814 16575 leader_election.cc:302] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:29.223081 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:29.225567 20758 raft_consensus.cc:3037] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:29.225972 16575 leader_election.cc:302] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:29.226778 20979 raft_consensus.cc:2781] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:29.142849 16701 leader_election.cc:302] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:29.227236 20979 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
W20260810 18:52:29.227634 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 84864.7us
I20260810 18:52:29.227809 20797 raft_consensus.cc:2781] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:29.228331 16701 leader_election.cc:302] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:29.228358 20797 raft_consensus.cc:686] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:29.229678 20797 consensus_queue.cc:237] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:29.227643 20979 raft_consensus.cc:3037] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:29.240175 16746 raft_consensus.cc:1226] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:29.242847 16746 raft_consensus.cc:1226] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:29.245402 20927 raft_consensus.cc:2726] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:52:29.250887 20979 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:29.251358 17780 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 7.
I20260810 18:52:29.255272 20942 raft_consensus.cc:2781] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:29.255312 21030 consensus_queue.cc:1046] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260810 18:52:29.260411 16749 raft_consensus.cc:1226] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: []
W20260810 18:52:29.265700 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 175621us
I20260810 18:52:29.265759 16575 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:29.268373 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:29.274448 16743 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:29.268826 20942 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:29.269541 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:29.270272 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:29.275974 16747 raft_consensus.cc:2396] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 8 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260810 18:52:29.224179 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:29.276351 16505 raft_consensus.cc:2379] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 7: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:29.271579 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:29.272143 16752 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd"
dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
from {username='slave'} at 127.0.0.1:43738
I20260810 18:52:29.272221 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:29.276690 16500 raft_consensus.cc:3037] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:29.272897 16750 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8cbe74806d8349e690b392275de70292"
dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
from {username='slave'} at 127.0.0.1:43738
I20260810 18:52:29.277045 16752 raft_consensus.cc:482] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:52:29.273939 21030 raft_consensus.cc:3032] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Stepping down as leader of term 6
I20260810 18:52:29.270746 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:29.277453 16752 raft_consensus.cc:3037] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:29.277469 16750 raft_consensus.cc:482] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:52:29.276835 20942 raft_consensus.cc:3037] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:29.280049 16750 raft_consensus.cc:3037] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:29.278505 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:29.286337 16695 leader_election.cc:302] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:29.279101 21030 raft_consensus.cc:729] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:29.291028 16752 raft_consensus.cc:504] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:29.298060 16695 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:29.291651 16507 raft_consensus.cc:1226] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:29.292214 16740 raft_consensus.cc:1226] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:29.298643 20994 consensus_queue.cc:1046] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260810 18:52:29.299799 16749 raft_consensus.cc:1226] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:29.303321 16746 raft_consensus.cc:1226] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 LEADER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:29.301414 20927 raft_consensus.cc:2726] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:29.291337 20912 raft_consensus.cc:2726] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:52:29.304469 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:29.296025 21030 consensus_queue.cc:260] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:29.306808 21030 raft_consensus.cc:3037] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:29.307770 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:29.309499 16695 leader_election.cc:302] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:29.310253 20912 raft_consensus.cc:3032] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Stepping down as leader of term 6
I20260810 18:52:29.310591 20912 raft_consensus.cc:729] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:29.297209 16750 raft_consensus.cc:504] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:29.311437 20912 consensus_queue.cc:260] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:29.312668 20912 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:29.313321 16750 leader_election.cc:288] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:29.314917 16743 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 7.
I20260810 18:52:29.291169 16500 raft_consensus.cc:2454] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 8.
I20260810 18:52:29.312587 20942 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:29.323489 17724 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:29.324726 16752 leader_election.cc:288] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:29.330394 16633 raft_consensus.cc:1226] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:29.333513 16575 leader_election.cc:302] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:29.333707 17724 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:29.335990 20945 raft_consensus.cc:2781] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:29.336529 20945 raft_consensus.cc:686] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:29.337118 20694 consensus_queue.cc:1046] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260810 18:52:29.337432 20945 consensus_queue.cc:237] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:29.339118 20694 raft_consensus.cc:3032] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 7 LEADER]: Stepping down as leader of term 7
I20260810 18:52:29.339497 20694 raft_consensus.cc:729] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:29.340373 20694 consensus_queue.cc:260] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:29.320665 20940 raft_consensus.cc:2726] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:29.331904 20994 consensus_queue.cc:1046] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.014s
I20260810 18:52:29.346215 20694 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:29.327745 21028 raft_consensus.cc:2726] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:29.347638 17724 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:29.369094 16575 leader_election.cc:302] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:29.374660 16624 raft_consensus.cc:1226] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:29.376802 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.380738 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.382979 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.384066 20979 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:29.345140 20758 raft_consensus.cc:2726] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:29.384891 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.385901 16505 raft_consensus.cc:2354] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 6. Current term is 7.
I20260810 18:52:29.386142 20942 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:29.388016 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 44401.5us
I20260810 18:52:29.368966 17724 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:29.390086 17724 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:29.392567 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.395306 17724 leader_election.cc:398] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 election: Vote denied by peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791) with higher term. Message: Invalid argument: T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for earlier term 6. Current term is 7.
I20260810 18:52:29.395995 16507 raft_consensus.cc:2379] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 8: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:29.398144 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.371177 20459 raft_consensus.cc:2781] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:29.347556 21028 raft_consensus.cc:3032] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Stepping down as leader of term 6
I20260810 18:52:29.399864 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
W20260810 18:52:29.401582 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 78569.4us
I20260810 18:52:29.404450 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.405236 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:29.420075 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:29.420495 16695 leader_election.cc:302] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:29.406800 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:29.411577 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
W20260810 18:52:29.412583 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 43584.4us
I20260810 18:52:29.418941 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.406080 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:29.425318 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:29.426666 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.428089 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:29.430392 16740 raft_consensus.cc:1226] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:29.429356 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.431980 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.432170 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:29.433326 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:29.433881 16505 raft_consensus.cc:2454] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:29.387764 20966 raft_consensus.cc:2726] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260810 18:52:29.420173 16506 raft_consensus.cc:2454] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:29.396633 21017 raft_consensus.cc:2726] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260810 18:52:29.400029 20948 raft_consensus.cc:2781] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:29.443049 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:29.412272 20758 raft_consensus.cc:3037] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:29.442423 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:29.441510 16575 leader_election.cc:302] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:29.445477 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.436350 20459 raft_consensus.cc:686] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 6 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:29.419083 21028 raft_consensus.cc:729] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
W20260810 18:52:29.454674 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 48440.3us
I20260810 18:52:29.452323 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.455401 17724 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:29.448258 16507 raft_consensus.cc:2454] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 7.
I20260810 18:52:29.440158 21050 raft_consensus.cc:2726] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:29.419378 19587 consensus_queue.cc:1046] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20260810 18:52:29.475618 16575 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:29.479913 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:29.458696 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:29.455936 16504 raft_consensus.cc:2454] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:29.453562 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:29.474046 20927 raft_consensus.cc:2781] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:29.456262 20948 raft_consensus.cc:686] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 6 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:29.454473 21017 consensus_queue.cc:1046] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260810 18:52:29.499440 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.500523 16504 raft_consensus.cc:1226] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:29.501966 20825 consensus_queue.cc:1046] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.100s
W20260810 18:52:29.504282 21044 log.cc:924] Time spent T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.082s user 0.001s sys 0.000s
I20260810 18:52:29.511037 16575 leader_election.cc:302] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:29.491019 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:29.515142 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.515539 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:29.516613 17770 raft_consensus.cc:2379] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 7: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:29.518528 20902 raft_consensus.cc:3032] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Stepping down as leader of term 4
I20260810 18:52:29.519053 20902 raft_consensus.cc:729] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:29.520102 20902 consensus_queue.cc:260] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:29.502795 20966 raft_consensus.cc:2726] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260810 18:52:29.477641 20894 consensus_queue.cc:1035] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.032s
I20260810 18:52:29.521654 20902 raft_consensus.cc:3037] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:29.523571 17780 raft_consensus.cc:2379] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 7: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:52:29.531659 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:29.567334 16750 raft_consensus.cc:1226] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 1. Current term is 3. Ops: []
I20260810 18:52:29.533404 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.538265 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.540421 21060 raft_consensus.cc:3032] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:29.541249 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.542162 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.543440 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.544229 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.580130 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:29.544970 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.545730 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:29.580906 16504 raft_consensus.cc:3032] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:29.546442 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.581260 16505 raft_consensus.cc:2454] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:29.555979 20948 consensus_queue.cc:237] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:29.558086 16695 leader_election.cc:302] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:29.582599 17786 raft_consensus.cc:2379] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 7: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:52:29.559217 16752 raft_consensus.cc:1226] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:29.560729 16740 raft_consensus.cc:1226] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:29.573943 16750 raft_consensus.cc:1226] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:29.577869 16743 raft_consensus.cc:1226] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:29.583846 20854 raft_consensus.cc:3032] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:29.584357 20854 raft_consensus.cc:729] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:29.582005 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:29.581236 16504 raft_consensus.cc:729] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:29.586958 16507 raft_consensus.cc:3032] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:52:29.589460 16507 raft_consensus.cc:729] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:29.563766 20459 consensus_queue.cc:237] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:29.603744 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:29.564234 20979 raft_consensus.cc:2726] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:29.586237 16504 consensus_queue.cc:260] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:29.606098 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 48120.2us
I20260810 18:52:29.606678 16504 raft_consensus.cc:3037] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
W20260810 18:52:29.605347 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 94418.4us
I20260810 18:52:29.608202 16575 leader_election.cc:302] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:29.609220 16575 leader_election.cc:302] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:29.610060 16575 leader_election.cc:302] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:29.611135 20898 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:29.611627 20898 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:29.613561 20898 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:29.595960 16507 consensus_queue.cc:260] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:29.614416 16507 raft_consensus.cc:3037] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 1 FOLLOWER]: Advancing to term 3
I20260810 18:52:29.507457 21028 consensus_queue.cc:260] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:29.579388 21060 raft_consensus.cc:729] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:29.619678 20979 raft_consensus.cc:2726] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:29.620348 20979 raft_consensus.cc:2726] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:29.620951 20979 raft_consensus.cc:2781] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:29.621356 20979 raft_consensus.cc:482] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:29.621721 20979 raft_consensus.cc:3037] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:29.604511 20902 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:29.622609 20902 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:29.624334 20854 consensus_queue.cc:260] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:29.625756 20854 raft_consensus.cc:3037] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:29.627478 20952 consensus_queue.cc:1046] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:52:29.628562 20952 raft_consensus.cc:3032] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:52:29.628917 20952 raft_consensus.cc:729] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:29.629735 20952 consensus_queue.cc:260] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:29.633284 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:29.602658 20942 raft_consensus.cc:2726] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:29.634148 17771 raft_consensus.cc:3037] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:29.634629 16502 raft_consensus.cc:2454] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:29.635615 17786 raft_consensus.cc:1226] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:29.643334 16504 raft_consensus.cc:2454] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:29.645931 16507 raft_consensus.cc:2454] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:29.596478 20894 raft_consensus.cc:2781] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:29.646879 16450 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:29.647161 16695 leader_election.cc:398] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Vote denied by peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791) with higher term. Message: Invalid argument: T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 6. Current term is 7.
I20260810 18:52:29.647508 16695 leader_election.cc:401] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
W20260810 18:52:29.647851 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 215592us
I20260810 18:52:29.660879 16740 raft_consensus.cc:1261] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260810 18:52:29.653759 16743 raft_consensus.cc:1261] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260810 18:52:29.650728 16450 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:29.667784 17786 raft_consensus.cc:1226] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:29.667894 16500 raft_consensus.cc:2454] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
W20260810 18:52:29.672214 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 236344us
I20260810 18:52:29.675645 16623 raft_consensus.cc:1226] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:29.648603 20894 raft_consensus.cc:482] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:29.672667 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 62649.9us
I20260810 18:52:29.663260 20952 raft_consensus.cc:3037] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:29.673595 21028 raft_consensus.cc:3037] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:29.646463 19907 consensus_queue.cc:1046] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260810 18:52:29.671240 20942 raft_consensus.cc:2726] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791) with higher term. Message: Invalid argument: T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 6. Current term is 7.
I20260810 18:52:29.691747 20247 raft_consensus.cc:2726] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260810 18:52:29.693800 20902 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:29.669533 20927 raft_consensus.cc:686] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:29.701239 16575 leader_election.cc:302] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:29.712252 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:29.660358 21004 consensus_queue.cc:1046] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20260810 18:52:29.695971 20894 raft_consensus.cc:3037] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
W20260810 18:52:29.719588 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 45293.2us
I20260810 18:52:29.720625 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:29.723758 16623 raft_consensus.cc:2454] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
I20260810 18:52:29.721748 16636 raft_consensus.cc:1226] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:29.670516 21017 consensus_queue.cc:1035] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20260810 18:52:29.696758 20825 raft_consensus.cc:2726] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260810 18:52:29.731583 21070 consensus_queue.cc:1046] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.019s
I20260810 18:52:29.743013 21070 raft_consensus.cc:3032] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:29.743372 21070 raft_consensus.cc:729] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:29.744096 21070 consensus_queue.cc:260] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:29.745368 21070 raft_consensus.cc:3037] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:29.746326 20894 raft_consensus.cc:504] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:29.687764 21069 consensus_queue.cc:1046] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.065s
I20260810 18:52:29.747572 21060 consensus_queue.cc:260] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:29.748265 21069 raft_consensus.cc:3032] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 LEADER]: Stepping down as leader of term 6
I20260810 18:52:29.748757 21069 raft_consensus.cc:729] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:29.748888 21060 raft_consensus.cc:3037] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:29.749838 21069 consensus_queue.cc:260] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:29.736249 20979 raft_consensus.cc:504] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:29.737695 20902 raft_consensus.cc:3032] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 6 LEADER]: Stepping down as leader of term 6
I20260810 18:52:29.672650 20966 consensus_queue.cc:1035] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20260810 18:52:29.731786 20927 consensus_queue.cc:237] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:29.741617 17783 raft_consensus.cc:2454] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 8.
W20260810 18:52:29.764628 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 67641.8us
I20260810 18:52:29.770119 16575 leader_election.cc:302] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:29.759505 21069 raft_consensus.cc:3037] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:29.755472 20902 raft_consensus.cc:729] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:29.774859 16575 leader_election.cc:302] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:29.783555 21061 raft_consensus.cc:2781] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:52:29.786310 20247 consensus_queue.cc:1046] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:52:29.787520 20247 consensus_queue.cc:1046] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:52:29.788232 20247 consensus_queue.cc:1225] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer 980f343dd1fd4e049143f8b9d9f2f1be log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260810 18:52:29.789225 20247 raft_consensus.cc:3032] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:52:29.789707 20247 raft_consensus.cc:729] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:29.789031 20902 consensus_queue.cc:260] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:29.790656 20902 raft_consensus.cc:3037] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:29.791126 21057 raft_consensus.cc:3032] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Stepping down as leader of term 4
I20260810 18:52:29.791532 21057 raft_consensus.cc:729] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:29.792349 21057 consensus_queue.cc:260] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:29.793475 21057 raft_consensus.cc:3037] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:29.787828 21038 consensus_queue.cc:1046] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:52:29.801909 17771 raft_consensus.cc:2454] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 8.
I20260810 18:52:29.805361 16575 leader_election.cc:302] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:29.810525 21035 raft_consensus.cc:3032] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:52:29.810711 20894 leader_election.cc:288] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:29.811049 21035 raft_consensus.cc:729] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:29.812038 21035 consensus_queue.cc:260] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:29.813535 21035 raft_consensus.cc:3037] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
W20260810 18:52:29.816356 21063 log.cc:924] Time spent T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.115s user 0.000s sys 0.001s
W20260810 18:52:29.817210 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 179346us
I20260810 18:52:29.818557 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.820420 21078 raft_consensus.cc:3032] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Stepping down as leader of term 4
I20260810 18:52:29.820573 20894 raft_consensus.cc:2726] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:29.820833 21078 raft_consensus.cc:729] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
W20260810 18:52:29.822055 21084 log.cc:924] Time spent T 00000000000000000000000000000000 P 0975c6fa18274f2baee7fb3615d5b985: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20260810 18:52:29.821796 21078 consensus_queue.cc:260] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:29.824554 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.825316 16500 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 8.
I20260810 18:52:29.825460 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:29.829250 16695 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:29.829620 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.830417 16504 raft_consensus.cc:2379] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 3: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:29.830415 16695 leader_election.cc:302] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:29.831849 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:29.834084 16695 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:29.836637 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:29.838277 16695 leader_election.cc:302] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:29.835378 20942 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:29.835522 20247 consensus_queue.cc:260] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:29.842000 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.842571 20247 raft_consensus.cc:3037] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:29.843166 21050 raft_consensus.cc:2726] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260810 18:52:29.844946 16502 raft_consensus.cc:1261] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260810 18:52:29.845259 20758 raft_consensus.cc:2726] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:29.839895 21030 raft_consensus.cc:2726] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:52:29.848992 20979 leader_election.cc:288] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:29.852746 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.853615 16500 raft_consensus.cc:2454] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:52:29.855626 16507 raft_consensus.cc:2454] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
I20260810 18:52:29.856650 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.826287 21078 raft_consensus.cc:3037] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:29.856266 16695 leader_election.cc:302] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:29.858201 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:29.856946 16495 raft_consensus.cc:2454] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:52:29.861351 16506 raft_consensus.cc:2379] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 6: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:52:29.866403 20994 raft_consensus.cc:2726] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:29.868651 16632 raft_consensus.cc:1226] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:29.870334 16622 raft_consensus.cc:1226] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:29.872094 16636 raft_consensus.cc:1261] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260810 18:52:29.873754 16624 raft_consensus.cc:1261] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260810 18:52:29.876075 21059 consensus_queue.cc:1046] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:52:29.877149 21059 consensus_queue.cc:1046] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:52:29.878106 21059 raft_consensus.cc:3032] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:52:29.878407 21059 raft_consensus.cc:729] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:29.879282 20858 raft_consensus.cc:3032] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:52:29.879778 20858 raft_consensus.cc:729] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:29.880841 20942 raft_consensus.cc:2726] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260810 18:52:29.881793 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:29.880858 20858 consensus_queue.cc:260] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:29.883183 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.883818 21081 raft_consensus.cc:2726] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:29.883769 20858 raft_consensus.cc:3037] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:29.883181 21083 raft_consensus.cc:2726] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:52:29.904261 16745 raft_consensus.cc:1261] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260810 18:52:29.904815 16502 raft_consensus.cc:1226] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 7. Current term is 8. Ops: []
I20260810 18:52:29.905725 16504 raft_consensus.cc:2454] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:52:29.887774 16695 leader_election.cc:302] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:29.882583 16752 raft_consensus.cc:2454] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:29.893764 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:29.906795 16750 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 8.
I20260810 18:52:29.891741 21081 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:29.901990 16749 raft_consensus.cc:1261] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260810 18:52:29.907914 16695 leader_election.cc:302] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:29.913098 16695 leader_election.cc:302] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:29.909811 16507 raft_consensus.cc:2454] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
I20260810 18:52:29.909865 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:29.915729 16695 leader_election.cc:302] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:29.859241 21050 raft_consensus.cc:2726] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:52:29.938515 16504 raft_consensus.cc:1226] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:29.938876 16695 leader_election.cc:302] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:29.912133 21083 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:29.940903 20940 raft_consensus.cc:2726] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:29.940595 21083 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:29.941792 16695 leader_election.cc:302] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:29.943317 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:29.943750 16448 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:29.945214 21057 raft_consensus.cc:2781] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:29.938971 21055 raft_consensus.cc:2741] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260810 18:52:29.913296 20942 raft_consensus.cc:2726] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:52:29.918304 20994 raft_consensus.cc:2726] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:52:29.916440 21069 raft_consensus.cc:482] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:29.920419 21028 raft_consensus.cc:2726] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260810 18:52:29.949718 16695 leader_election.cc:302] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:29.916450 21061 raft_consensus.cc:686] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 4 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:29.951937 21050 raft_consensus.cc:2726] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:52:29.956898 16502 raft_consensus.cc:1226] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 6 LEADER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:29.957526 16695 leader_election.cc:302] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:29.959479 16695 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:29.961634 16495 raft_consensus.cc:1261] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260810 18:52:29.962067 16695 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:29.960857 19587 consensus_queue.cc:1046] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260810 18:52:29.949596 21057 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:29.961547 16507 raft_consensus.cc:2454] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
I20260810 18:52:29.964794 21087 consensus_queue.cc:1035] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20260810 18:52:29.920797 21081 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:29.967795 21081 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:29.969161 21057 raft_consensus.cc:3037] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:29.967666 19587 raft_consensus.cc:3032] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Stepping down as leader of term 4
I20260810 18:52:29.961236 20994 raft_consensus.cc:2726] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:29.970126 19587 raft_consensus.cc:729] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:29.957006 21069 raft_consensus.cc:504] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:29.971145 19587 consensus_queue.cc:260] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:29.972524 19587 raft_consensus.cc:3037] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:29.973192 21069 leader_election.cc:288] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:29.974511 21083 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:29.976112 16450 leader_election.cc:302] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:29.977031 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:29.952113 20902 consensus_queue.cc:1035] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20260810 18:52:29.980451 21038 raft_consensus.cc:2781] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260810 18:52:29.979517 16695 leader_election.cc:302] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:29.981999 16633 raft_consensus.cc:1226] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:29.981349 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:29.984373 16507 raft_consensus.cc:2379] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 6: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:29.985522 16695 leader_election.cc:302] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:29.961014 21028 raft_consensus.cc:2726] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:52:29.990542 16695 leader_election.cc:302] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:29.980930 21038 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:29.917140 21059 consensus_queue.cc:260] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:29.992205 21038 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:29.996304 19907 consensus_queue.cc:1035] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20260810 18:52:29.978155 21050 raft_consensus.cc:2726] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260810 18:52:29.992710 20854 consensus_queue.cc:1035] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.012s
I20260810 18:52:29.984655 21030 raft_consensus.cc:2781] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:29.988498 20994 raft_consensus.cc:2726] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260810 18:52:30.002100 21030 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:30.002544 21030 raft_consensus.cc:3037] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:30.002399 20825 consensus_queue.cc:1046] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:52:30.004261 21059 raft_consensus.cc:3037] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:30.008358 21057 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:30.014067 21057 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:29.988914 20942 raft_consensus.cc:2741] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260810 18:52:30.019032 20825 raft_consensus.cc:3032] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:52:30.020768 20825 raft_consensus.cc:729] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:30.021622 20825 consensus_queue.cc:260] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:30.023196 16507 raft_consensus.cc:1226] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 6. Current term is 7. Ops: []
W20260810 18:52:30.019961 21093 log.cc:924] Time spent T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20260810 18:52:30.026401 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:30.026949 16749 raft_consensus.cc:3037] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 7
W20260810 18:52:30.049151 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 58672.7us
I20260810 18:52:30.050130 16695 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:30.048729 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:30.051833 16622 raft_consensus.cc:3037] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:30.054131 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:30.054867 16507 raft_consensus.cc:2379] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 7: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:30.032311 20825 raft_consensus.cc:3037] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:30.048460 21028 raft_consensus.cc:2726] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:30.069842 21038 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:30.070182 21030 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:30.072425 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:30.072707 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:30.073240 16495 raft_consensus.cc:2379] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 7: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:30.074916 16573 leader_election.cc:302] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:30.076540 17724 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:30.076933 21070 raft_consensus.cc:3037] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:30.078029 17771 raft_consensus.cc:1261] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260810 18:52:30.079128 21053 consensus_queue.cc:1046] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260810 18:52:30.080366 21053 raft_consensus.cc:2781] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader election won for term 8
I20260810 18:52:30.080901 21053 raft_consensus.cc:686] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 8 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:30.082316 21062 raft_consensus.cc:3032] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 6 LEADER]: Stepping down as leader of term 6
I20260810 18:52:30.082682 21062 raft_consensus.cc:729] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:30.083777 16636 raft_consensus.cc:1261] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260810 18:52:30.076568 16573 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:30.085029 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:30.075415 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:30.085731 16504 raft_consensus.cc:2454] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:52:30.083580 21062 consensus_queue.cc:260] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:30.088675 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 283418us
I20260810 18:52:30.088961 16749 raft_consensus.cc:2454] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
I20260810 18:52:30.089434 16575 leader_election.cc:302] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:30.090142 16450 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:30.091689 16447 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:30.092036 16504 raft_consensus.cc:1261] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260810 18:52:30.093250 21030 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:30.095422 21090 log.cc:924] Time spent T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.113s user 0.000s sys 0.001s
I20260810 18:52:30.095504 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:30.096910 21089 consensus_queue.cc:1035] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260810 18:52:30.098721 16501 raft_consensus.cc:1261] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260810 18:52:30.105822 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 55767us
I20260810 18:52:30.107299 21069 raft_consensus.cc:2726] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:30.107966 21069 consensus_queue.cc:1035] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:52:30.102355 21089 raft_consensus.cc:2726] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:30.110424 16575 leader_election.cc:302] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:30.111346 21081 raft_consensus.cc:2726] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:30.111951 21081 consensus_queue.cc:1035] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 18:52:30.112141 16573 leader_election.cc:302] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:30.099810 20942 raft_consensus.cc:2726] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260810 18:52:30.113233 21081 raft_consensus.cc:2781] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:30.090830 21061 consensus_queue.cc:237] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:30.113757 16573 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:30.114532 20758 raft_consensus.cc:3037] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:30.095530 21057 raft_consensus.cc:2781] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election won for term 7
I20260810 18:52:30.104563 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:30.117089 16506 raft_consensus.cc:2379] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 6: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:52:30.117061 21038 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:30.118829 16573 leader_election.cc:302] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:30.118899 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:30.119611 16740 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:30.113626 21081 raft_consensus.cc:482] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:30.120573 16507 raft_consensus.cc:2454] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:52:30.111960 16575 leader_election.cc:302] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:30.125947 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:30.129069 16502 raft_consensus.cc:2454] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:30.123517 21057 raft_consensus.cc:686] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:30.131853 16622 raft_consensus.cc:2454] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
I20260810 18:52:30.123216 21081 raft_consensus.cc:3037] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:30.139919 16575 leader_election.cc:302] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:30.135813 16573 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:30.128492 21062 raft_consensus.cc:3037] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:30.138150 16447 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:30.127914 20927 raft_consensus.cc:2781] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:30.081840 21053 consensus_queue.cc:237] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:30.151796 16447 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:30.152639 16695 leader_election.cc:302] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:30.154870 16575 leader_election.cc:302] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:30.148110 21106 log.cc:924] Time spent T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20260810 18:52:30.155195 16447 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:30.160161 20459 consensus_queue.cc:1035] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260810 18:52:30.160055 21057 consensus_queue.cc:237] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:30.163582 16575 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:30.165741 16573 leader_election.cc:302] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:30.167184 16701 leader_election.cc:302] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:30.168316 16701 leader_election.cc:302] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:30.168774 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:30.170684 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:30.170742 16695 leader_election.cc:302] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:30.152083 20927 raft_consensus.cc:482] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:30.172263 16622 raft_consensus.cc:3037] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 1 FOLLOWER]: Advancing to term 3
I20260810 18:52:30.172571 20927 raft_consensus.cc:3037] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:30.157097 20825 raft_consensus.cc:2726] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260810 18:52:30.162343 20858 raft_consensus.cc:2781] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader election won for term 8
I20260810 18:52:30.175441 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:30.155426 20979 raft_consensus.cc:2741] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20260810 18:52:30.158737 21069 raft_consensus.cc:2726] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:30.153293 21083 raft_consensus.cc:2726] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260810 18:52:30.176609 16632 raft_consensus.cc:2379] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 6: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:30.178470 21030 raft_consensus.cc:2781] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:30.179126 21055 raft_consensus.cc:2781] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:30.181766 16750 raft_consensus.cc:2354] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 7. Current term is 8.
I20260810 18:52:30.151648 21061 raft_consensus.cc:2726] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:30.153000 21035 raft_consensus.cc:2726] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260810 18:52:30.151552 21104 raft_consensus.cc:2781] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:30.187258 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:30.188860 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:30.193665 16750 raft_consensus.cc:1226] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 7. Current term is 8. Ops: [5.3-7.4]
I20260810 18:52:30.194636 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:30.195374 16623 raft_consensus.cc:3037] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:30.194710 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:52:30.268479 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 97820.6us
I20260810 18:52:30.269130 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:30.176040 20858 raft_consensus.cc:686] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 8 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:30.169973 20942 raft_consensus.cc:2726] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:30.271556 16752 raft_consensus.cc:2454] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:52:30.272042 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:30.271807 16628 raft_consensus.cc:1261] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260810 18:52:30.181157 20894 raft_consensus.cc:2781] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader election won for term 7
I20260810 18:52:30.275754 16623 raft_consensus.cc:2454] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 8.
W20260810 18:52:30.276376 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 87293.7us
I20260810 18:52:30.277535 16622 raft_consensus.cc:2454] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:30.188966 20979 raft_consensus.cc:2726] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260810 18:52:30.163865 21089 consensus_queue.cc:1035] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.110s
I20260810 18:52:30.192277 21030 raft_consensus.cc:686] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:30.270860 21055 raft_consensus.cc:686] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:30.291643 16695 leader_election.cc:302] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:30.191504 21028 raft_consensus.cc:2726] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:52:30.321630 16622 raft_consensus.cc:1261] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260810 18:52:30.324708 16695 leader_election.cc:302] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:30.325531 21028 raft_consensus.cc:2726] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260810 18:52:30.326278 21110 log.cc:924] Time spent T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.129s user 0.001s sys 0.000s
I20260810 18:52:30.335348 16622 raft_consensus.cc:1261] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260810 18:52:30.290200 21104 raft_consensus.cc:686] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:30.336365 21030 consensus_queue.cc:237] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:30.338248 21070 raft_consensus.cc:2726] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:52:30.289281 20459 consensus_queue.cc:1035] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.090s
I20260810 18:52:30.291862 20758 raft_consensus.cc:2726] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:30.346197 16623 raft_consensus.cc:3037] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:30.392750 21081 raft_consensus.cc:504] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:30.395478 16632 raft_consensus.cc:1261] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260810 18:52:30.395543 21104 consensus_queue.cc:237] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:30.346526 16740 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 9.
I20260810 18:52:30.348165 20858 consensus_queue.cc:237] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:30.399655 16743 raft_consensus.cc:1226] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260810 18:52:30.351889 20927 raft_consensus.cc:504] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:30.428781 20927 leader_election.cc:288] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:30.357455 16575 leader_election.cc:398] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 election: Vote denied by peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171) with higher term. Message: Invalid argument: T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 7. Current term is 8.
I20260810 18:52:30.429530 16575 leader_election.cc:401] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
W20260810 18:52:30.430363 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 75480.2us
I20260810 18:52:30.359941 21055 consensus_queue.cc:237] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:30.369329 16506 raft_consensus.cc:1261] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260810 18:52:30.370379 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 204731us
I20260810 18:52:30.433414 16573 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:30.303339 20894 raft_consensus.cc:686] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 7 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:30.436637 16575 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:30.437763 16575 leader_election.cc:302] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:30.438587 16573 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:30.384779 21112 log.cc:924] Time spent T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20260810 18:52:30.349534 16701 leader_election.cc:302] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
W20260810 18:52:30.440641 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 91206.9us
I20260810 18:52:30.441850 16701 leader_election.cc:302] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
W20260810 18:52:30.421911 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 97297.7us
I20260810 18:52:30.425153 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260810 18:52:30.445647 16695 leader_election.cc:302] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:30.403473 16623 raft_consensus.cc:1261] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260810 18:52:30.394898 21081 leader_election.cc:288] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:30.311247 21095 consensus_queue.cc:1035] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.139s
W20260810 18:52:30.446592 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 133194us
I20260810 18:52:30.439855 16450 leader_election.cc:302] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:30.451762 16695 leader_election.cc:302] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:30.458235 16573 leader_election.cc:302] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:30.435957 21028 raft_consensus.cc:2781] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:52:30.460050 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:30.463840 16752 raft_consensus.cc:2379] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 4: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:30.418349 20966 consensus_queue.cc:1035] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.057s
I20260810 18:52:30.466977 16740 raft_consensus.cc:1261] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260810 18:52:30.467722 16573 leader_election.cc:302] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:30.458124 20758 raft_consensus.cc:3037] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:30.470250 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:30.454716 20979 raft_consensus.cc:2726] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:30.455631 20942 raft_consensus.cc:2781] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:30.453207 20940 raft_consensus.cc:2781] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader election won for term 8
I20260810 18:52:30.471323 16745 raft_consensus.cc:2379] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 4: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:30.443179 21060 consensus_queue.cc:1035] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.026s
I20260810 18:52:30.479389 20912 raft_consensus.cc:2781] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:52:30.453354 21070 consensus_queue.cc:1046] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.019s
I20260810 18:52:30.462206 21050 raft_consensus.cc:2781] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader election won for term 3
W20260810 18:52:30.493319 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 53566.6us
I20260810 18:52:30.462337 20927 raft_consensus.cc:2781] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:30.466325 21081 raft_consensus.cc:2726] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:30.453012 20459 consensus_queue.cc:1035] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260810 18:52:30.511862 21116 raft_consensus.cc:2726] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171) with higher term. Message: Invalid argument: T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 7. Current term is 8.
I20260810 18:52:30.515465 21070 raft_consensus.cc:3032] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 LEADER]: Stepping down as leader of term 7
I20260810 18:52:30.516008 21070 raft_consensus.cc:729] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:30.480506 21083 raft_consensus.cc:2781] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:52:30.496225 21061 raft_consensus.cc:2781] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election won for term 5
W20260810 18:52:30.559147 21124 log.cc:924] Time spent T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.062s user 0.000s sys 0.001s
I20260810 18:52:30.472146 21069 consensus_queue.cc:1046] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20260810 18:52:30.504030 20902 raft_consensus.cc:2781] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Leader election won for term 9
I20260810 18:52:30.506063 20927 raft_consensus.cc:482] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:30.527303 21116 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:30.558938 20994 consensus_queue.cc:1046] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.010s
I20260810 18:52:30.498656 20979 consensus_queue.cc:1035] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.109s
I20260810 18:52:30.635736 20927 raft_consensus.cc:3037] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
W20260810 18:52:30.652961 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 135314us
I20260810 18:52:30.636173 20902 raft_consensus.cc:686] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 9 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:30.638433 21028 raft_consensus.cc:686] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:30.608108 21083 raft_consensus.cc:686] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 4 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
W20260810 18:52:30.713657 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 246014us
I20260810 18:52:30.520215 21122 consensus_queue.cc:1035] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.187s
I20260810 18:52:30.717643 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:30.686511 21078 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:30.689311 20912 raft_consensus.cc:686] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:30.729521 21061 raft_consensus.cc:686] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:30.735509 16502 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202"
dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
mode: GRACEFUL
new_leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
from {username='slave'} at 127.0.0.1:51572
I20260810 18:52:30.774619 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:30.706506 21062 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:30.708863 20942 raft_consensus.cc:686] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
W20260810 18:52:30.776548 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 44443.5us
I20260810 18:52:30.783087 16502 raft_consensus.cc:595] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Received request to transfer leadership to TS 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:30.786540 16502 raft_consensus.cc:603] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260810 18:52:30.639724 21070 consensus_queue.cc:260] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:30.642894 21116 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:30.774643 21050 raft_consensus.cc:686] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:30.785457 21138 raft_consensus.cc:482] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:30.640502 20894 consensus_queue.cc:237] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:30.816008 16500 raft_consensus.cc:2454] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:52:30.818449 16573 leader_election.cc:302] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:30.736807 21078 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:30.823613 21135 log.cc:924] Time spent T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.129s user 0.001s sys 0.000s
I20260810 18:52:30.819509 20912 consensus_queue.cc:237] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:30.824398 20942 consensus_queue.cc:237] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:30.825256 21126 log.cc:924] Time spent T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.273s user 0.000s sys 0.001s
I20260810 18:52:30.807775 21069 raft_consensus.cc:482] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:30.808900 21070 raft_consensus.cc:3037] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Advancing to term 8
W20260810 18:52:30.843241 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 373474us
I20260810 18:52:30.790297 20902 consensus_queue.cc:237] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:30.793212 21062 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:30.809167 21064 consensus_queue.cc:1035] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.325s
I20260810 18:52:30.839126 20979 raft_consensus.cc:2781] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election won for term 4
W20260810 18:52:30.896413 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 115971us
I20260810 18:52:30.901162 16743 raft_consensus.cc:1261] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260810 18:52:30.903110 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:30.810981 21133 consensus_queue.cc:1035] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.350s
I20260810 18:52:30.815112 21138 raft_consensus.cc:504] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:30.820811 21028 consensus_queue.cc:237] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:30.841552 21069 raft_consensus.cc:504] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:30.826342 21061 consensus_queue.cc:237] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:30.910702 20979 raft_consensus.cc:686] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
W20260810 18:52:30.950304 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 50750.2us
I20260810 18:52:30.873020 21083 consensus_queue.cc:237] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:30.948889 21154 raft_consensus.cc:482] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:30.948894 21162 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:30.907058 21050 consensus_queue.cc:237] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:31.062239 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:31.045250 21149 log.cc:924] Time spent T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.226s user 0.001s sys 0.000s
I20260810 18:52:31.044442 21160 raft_consensus.cc:3032] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:31.074184 16695 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:30.949132 21163 consensus_queue.cc:1035] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.140s
I20260810 18:52:31.163940 16624 raft_consensus.cc:1261] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260810 18:52:31.167220 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:31.168145 16507 raft_consensus.cc:2454] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 8.
I20260810 18:52:31.171949 16573 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:31.175742 21174 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:31.078377 21160 raft_consensus.cc:729] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:31.172571 16636 raft_consensus.cc:1261] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260810 18:52:31.084805 21151 raft_consensus.cc:482] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:31.249547 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:31.250344 16504 raft_consensus.cc:2454] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:31.091950 21080 raft_consensus.cc:482] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:31.164109 21084 log.cc:924] Time spent T 00000000000000000000000000000000 P 0975c6fa18274f2baee7fb3615d5b985: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20260810 18:52:31.251766 16573 leader_election.cc:302] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:31.251817 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 68996.3us
I20260810 18:52:31.253214 16500 raft_consensus.cc:1261] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260810 18:52:31.296702 16502 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "681439c754d942b7847d73744abbe269"
dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
mode: GRACEFUL
new_leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
from {username='slave'} at 127.0.0.1:51572
I20260810 18:52:31.045467 21154 raft_consensus.cc:504] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:31.297509 16502 raft_consensus.cc:595] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Received request to transfer leadership to TS 0397d760589d40d3b92ef141cdfc2dbb
W20260810 18:52:31.298694 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 224610us
I20260810 18:52:31.304203 16502 raft_consensus.cc:603] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260810 18:52:31.312712 16495 consensus_meta.cc:287] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.585s user 0.009s sys 0.001s
I20260810 18:52:31.313205 16495 raft_consensus.cc:2454] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:52:31.253805 20825 raft_consensus.cc:482] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:31.247610 21185 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:31.255489 21062 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:31.328655 16636 raft_consensus.cc:1261] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260810 18:52:31.059355 21162 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:31.176303 21174 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:31.318012 21059 raft_consensus.cc:482] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:31.317049 21116 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:31.341132 21172 raft_consensus.cc:2781] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260810 18:52:31.161922 20979 consensus_queue.cc:237] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:31.327904 20912 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.502s user 0.007s sys 0.034s
I20260810 18:52:31.328874 21078 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:31.374281 21135 log.cc:924] Time spent T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
I20260810 18:52:31.329344 21163 raft_consensus.cc:482] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:31.363888 21138 leader_election.cc:288] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:31.293696 21080 raft_consensus.cc:504] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:31.411614 16740 raft_consensus.cc:1261] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260810 18:52:31.377605 21116 raft_consensus.cc:482] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:31.433698 21155 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:31.434401 21174 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:31.270972 21151 raft_consensus.cc:504] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:31.434283 21155 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:31.393661 20942 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.549s user 0.006s sys 0.011s
I20260810 18:52:31.440941 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:31.441627 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:31.341758 20825 raft_consensus.cc:504] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:31.400902 21191 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:31.341696 21185 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:31.400200 20912 raft_consensus.cc:2726] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260810 18:52:31.375275 21059 raft_consensus.cc:504] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:31.432579 20940 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.940s user 0.003s sys 0.009s
W20260810 18:52:31.530565 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 226023us
I20260810 18:52:31.543316 17771 raft_consensus.cc:2379] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 7: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:52:31.552685 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:31.549810 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 456374us
I20260810 18:52:31.565311 16495 raft_consensus.cc:2454] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 7.
W20260810 18:52:31.551088 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 299376us
I20260810 18:52:31.542958 17783 raft_consensus.cc:2454] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
I20260810 18:52:31.569072 16504 raft_consensus.cc:1261] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260810 18:52:31.442830 21174 raft_consensus.cc:482] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:31.568228 17724 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:31.459332 21181 log.cc:924] Time spent T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.207s user 0.000s sys 0.001s
W20260810 18:52:31.528858 21055 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.620s user 0.003s sys 0.002s
I20260810 18:52:31.608106 16501 raft_consensus.cc:1226] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:31.445617 21179 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:31.620949 16502 raft_consensus.cc:1261] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260810 18:52:31.622089 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:31.622991 16500 raft_consensus.cc:2379] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 4: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
W20260810 18:52:31.538961 20758 consensus_meta.cc:287] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 1.057s user 0.010s sys 0.002s
I20260810 18:52:31.460549 21192 raft_consensus.cc:2781] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:31.601673 16447 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:31.594312 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:31.459558 21193 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:31.520255 21138 raft_consensus.cc:482] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:31.459950 21176 raft_consensus.cc:482] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:31.542014 20940 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.052s user 0.005s sys 0.010s
I20260810 18:52:31.602679 16573 leader_election.cc:302] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:31.572391 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:31.608958 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:31.608099 16743 raft_consensus.cc:1261] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260810 18:52:31.611208 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:31.442338 21201 raft_consensus.cc:482] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:31.535401 21214 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:31.613484 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:31.617605 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:31.460357 21164 raft_consensus.cc:482] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:31.455145 21154 leader_election.cc:288] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:31.535255 21155 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:31.724367 21155 raft_consensus.cc:2781] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:31.724852 21155 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:31.725274 21155 raft_consensus.cc:3037] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Advancing to term 8
W20260810 18:52:31.682165 20927 consensus_meta.cc:287] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 1.038s user 0.007s sys 0.007s
I20260810 18:52:31.449119 21184 raft_consensus.cc:482] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:31.531502 21209 log.cc:924] Time spent T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.096s user 0.001s sys 0.000s
W20260810 18:52:31.542963 20942 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.696s user 0.006s sys 0.011s
I20260810 18:52:31.609053 21174 raft_consensus.cc:504] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:31.460525 21162 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:31.436766 21116 raft_consensus.cc:504] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:31.608513 21055 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.701s user 0.004s sys 0.002s
I20260810 18:52:31.579944 20979 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:31.608950 21078 raft_consensus.cc:482] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:31.693053 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:31.449074 21159 consensus_queue.cc:1035] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.322s
I20260810 18:52:31.429036 21163 raft_consensus.cc:504] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:31.634197 20758 raft_consensus.cc:2726] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:31.625655 20912 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:31.630316 21192 raft_consensus.cc:482] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:31.755249 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:31.564642 21191 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:31.676838 21083 raft_consensus.cc:482] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:31.757928 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:31.763628 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 161951us
I20260810 18:52:31.535615 21226 raft_consensus.cc:482] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:31.680897 21028 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:31.692523 21137 raft_consensus.cc:482] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:31.818164 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 58609.2us
I20260810 18:52:31.718024 20940 raft_consensus.cc:686] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:31.764457 20994 raft_consensus.cc:482] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:31.851990 16496 raft_consensus.cc:2454] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:31.631114 21179 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:31.727444 21214 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:31.716849 21176 raft_consensus.cc:504] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:31.851387 21192 raft_consensus.cc:3037] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:31.757897 21151 leader_election.cc:288] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:31.706387 21138 raft_consensus.cc:504] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:31.878089 16695 leader_election.cc:302] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:31.691236 21193 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:31.831806 21206 raft_consensus.cc:482] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:31.758658 20927 raft_consensus.cc:504] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:31.850455 21187 raft_consensus.cc:482] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:31.855769 21238 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:31.899370 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:31.897810 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 139495us
I20260810 18:52:31.744840 21164 raft_consensus.cc:504] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:31.830112 20979 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:31.833250 21078 raft_consensus.cc:504] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:31.903057 16506 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1555) took 1182 ms. Trace:
I20260810 18:52:31.850692 20912 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:31.925496 16621 raft_consensus.cc:1226] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:31.742539 21205 consensus_queue.cc:1035] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.192s
I20260810 18:52:31.929991 16749 raft_consensus.cc:2454] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
I20260810 18:52:31.871999 21080 leader_election.cc:288] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:31.887327 20825 leader_election.cc:288] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:31.871122 21255 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:31.925379 16506 rpcz_store.cc:276] 0810 18:52:30.715144 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:30.727568 (+ 12424us) service_pool.cc:227] Handling call
0810 18:52:30.780459 (+ 52891us) raft_consensus.cc:1457] Updating replica for 3 ops
0810 18:52:30.798443 (+ 17984us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:30.798492 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0810 18:52:30.799277 (+ 785us) log.cc:842] Serialized 78 byte log entry
0810 18:52:30.862554 (+ 63277us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:52:30.862611 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:30.862727 (+ 116us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:31.897430 (+1034703us) raft_consensus.cc:1671] finished
0810 18:52:31.897458 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:31.897662 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7486,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":93,"thread_start_us":70812,"threads_started":2,"wal-append.queue_time_us":275036}
I20260810 18:52:31.858443 21083 raft_consensus.cc:504] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:31.866768 21154 consensus_queue.cc:1046] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
I20260810 18:52:31.863385 21028 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:31.890064 21059 leader_election.cc:288] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:31.924703 21162 raft_consensus.cc:482] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:31.914754 21247 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:31.951735 21223 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:31.952387 21223 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:31.953398 21193 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:31.953958 21271 raft_consensus.cc:482] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:31.938640 16624 raft_consensus.cc:2454] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 7.
W20260810 18:52:31.944756 16448 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.687s user 0.006s sys 0.002s
I20260810 18:52:31.763767 21184 raft_consensus.cc:504] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:31.867137 20994 raft_consensus.cc:504] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:31.955214 16448 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.698s user 0.007s sys 0.002s
W20260810 18:52:31.962087 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 704859us
W20260810 18:52:31.950812 21261 log.cc:924] Time spent T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.117s user 0.001s sys 0.000s
I20260810 18:52:31.742278 21201 raft_consensus.cc:504] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:31.921146 21254 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:31.867838 21137 raft_consensus.cc:504] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:31.972827 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:31.926965 21238 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:31.982601 16622 raft_consensus.cc:2454] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
I20260810 18:52:31.989913 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:31.970083 21210 raft_consensus.cc:482] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:31.948891 21185 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:31.966534 21193 raft_consensus.cc:3037] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:31.920092 21187 raft_consensus.cc:504] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:31.947489 21245 consensus_queue.cc:1035] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.243s
I20260810 18:52:31.958559 21182 raft_consensus.cc:482] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:31.859742 21226 raft_consensus.cc:504] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:31.950124 21174 leader_election.cc:288] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:31.950124 21028 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:31.946787 21151 consensus_queue.cc:1035] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20260810 18:52:31.990764 16631 raft_consensus.cc:2454] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
W20260810 18:52:32.000211 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 80854.6us
W20260810 18:52:32.012739 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 410128us
W20260810 18:52:32.012125 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 134145us
W20260810 18:52:32.013461 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 412494us
W20260810 18:52:32.008265 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 159729us
I20260810 18:52:32.034272 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:32.035286 16752 raft_consensus.cc:2454] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:32.036710 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:31.979893 21268 raft_consensus.cc:2781] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:52:32.017244 21223 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:32.017329 20994 leader_election.cc:288] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:32.005360 21116 leader_election.cc:288] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:32.017874 21266 consensus_queue.cc:1035] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
W20260810 18:52:32.018433 21264 log.cc:924] Time spent T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
W20260810 18:52:32.018635 21070 consensus_meta.cc:287] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 1.174s user 0.015s sys 0.000s
I20260810 18:52:31.951103 21247 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:32.020258 21238 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:32.020869 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:31.918289 21206 raft_consensus.cc:504] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:32.021592 21237 log.cc:924] Time spent T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.316s user 0.000s sys 0.001s
I20260810 18:52:32.005602 21210 raft_consensus.cc:504] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:32.021909 20758 consensus_queue.cc:1035] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:52:32.022702 21201 leader_election.cc:288] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:31.950809 21162 raft_consensus.cc:504] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:31.994966 21265 raft_consensus.cc:2781] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:31.985181 21154 raft_consensus.cc:482] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:32.025471 21174 raft_consensus.cc:482] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:32.026005 21028 raft_consensus.cc:482] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:32.026844 21240 log.cc:924] Time spent T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.123s user 0.001s sys 0.000s
I20260810 18:52:31.954500 21271 raft_consensus.cc:504] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:32.049281 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:32.048365 21080 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:32.087249 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:31.949690 21244 consensus_queue.cc:1035] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.203s
I20260810 18:52:32.100875 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:31.957718 20940 consensus_queue.cc:237] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:32.101543 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:32.104071 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:32.121125 21176 leader_election.cc:288] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:32.121415 21179 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:31.948364 21255 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:32.206655 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:32.066350 21191 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:32.210007 16633 raft_consensus.cc:2454] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:32.220343 16631 raft_consensus.cc:2454] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:32.225793 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:32.221697 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:32.125226 21275 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:32.276633 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:32.126271 20912 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:32.102092 21268 raft_consensus.cc:686] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 4 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:32.222594 21265 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:32.156726 21070 raft_consensus.cc:482] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:32.122272 21247 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:32.126631 20942 raft_consensus.cc:482] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:32.108122 21163 leader_election.cc:288] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:32.015561 21182 raft_consensus.cc:504] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:32.181234 20825 consensus_queue.cc:1035] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.108s
W20260810 18:52:32.340852 16443 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.554s user 0.014s sys 0.027s
I20260810 18:52:32.066500 21155 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:32.347134 16742 raft_consensus.cc:2454] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:32.220474 21116 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:32.242192 21185 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:32.245427 21238 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:32.219940 21028 raft_consensus.cc:504] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:32.219311 21226 leader_election.cc:288] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:32.281754 21201 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:31.995471 21254 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:32.214416 21154 raft_consensus.cc:504] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:32.214604 21184 leader_election.cc:288] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:32.220168 21080 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:32.220850 21174 raft_consensus.cc:504] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:32.772596 16443 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.986s user 0.022s sys 0.045s
W20260810 18:52:32.476377 16575 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.006s sys 0.045s
I20260810 18:52:32.773028 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:32.773485 16575 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.819s user 0.011s sys 0.057s
I20260810 18:52:32.773783 16632 raft_consensus.cc:2454] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:32.754276 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:32.727660 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:32.778585 16633 raft_consensus.cc:2454] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:32.778738 16631 raft_consensus.cc:2454] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:32.723631 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:32.773041 16701 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.008s sys 0.004s
I20260810 18:52:32.486580 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:32.789417 16701 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.570s user 0.008s sys 0.004s
I20260810 18:52:32.282814 21191 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:32.772742 21138 leader_election.cc:288] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:32.453967 16745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43726 (request call id 1543) took 1623 ms. Trace:
I20260810 18:52:32.589461 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:32.347826 21265 raft_consensus.cc:3037] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Advancing to term 8
W20260810 18:52:32.866737 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 49096.7us
I20260810 18:52:32.858285 16622 raft_consensus.cc:2454] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
I20260810 18:52:32.372272 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:32.290936 21272 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:32.755810 21257 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:32.872701 16633 raft_consensus.cc:1261] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:52:32.753374 21228 raft_consensus.cc:482] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:32.883324 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:32.754014 21277 raft_consensus.cc:482] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:32.753144 21283 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:32.664270 21163 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:32.875180 21278 raft_consensus.cc:482] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:32.732117 21223 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:32.710714 21273 raft_consensus.cc:482] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:32.626788 21192 consensus_meta.cc:287] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.714s user 0.000s sys 0.010s
I20260810 18:52:32.857100 16745 rpcz_store.cc:276] 0810 18:52:30.830572 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:30.830731 (+ 159us) service_pool.cc:227] Handling call
0810 18:52:30.831088 (+ 357us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:30.831390 (+ 302us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:30.831419 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:30.831758 (+ 339us) log.cc:842] Serialized 26 byte log entry
0810 18:52:31.376092 (+544334us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:31.376223 (+ 131us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:31.376431 (+ 208us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:32.453685 (+1077254us) raft_consensus.cc:1671] finished
0810 18:52:32.453713 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:32.453892 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":91577,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":24730,"thread_start_us":477033,"threads_started":1,"wal-append.queue_time_us":714424}
I20260810 18:52:32.887212 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:32.373168 21214 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:32.753494 21239 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:32.818527 21137 leader_election.cc:288] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:32.890905 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:32.896399 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:32.488046 21164 leader_election.cc:288] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:32.620294 21246 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:32.902082 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:32.862525 21176 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:32.892613 17780 raft_consensus.cc:2454] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:32.906633 21283 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:32.928378 21210 leader_election.cc:288] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:32.930246 21283 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:32.882192 21184 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:32.931279 21184 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:32.933305 21184 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:32.923327 21278 raft_consensus.cc:504] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:32.936017 21278 leader_election.cc:288] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:32.898700 21187 leader_election.cc:288] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:32.943764 16450 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.592s user 0.004s sys 0.050s
W20260810 18:52:32.944115 16450 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.596s user 0.005s sys 0.051s
I20260810 18:52:32.896649 21267 raft_consensus.cc:482] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:32.351034 21070 raft_consensus.cc:504] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:32.908423 16747 raft_consensus.cc:2379] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 4: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:32.872251 20912 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:32.821659 21201 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:32.913312 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:32.904440 21163 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:32.917097 21296 raft_consensus.cc:482] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:32.917827 21214 raft_consensus.cc:482] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:32.918195 21239 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:32.375648 20942 raft_consensus.cc:504] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:32.924755 16632 raft_consensus.cc:2454] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:32.821251 21116 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:32.574522 21083 leader_election.cc:288] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:32.826035 21191 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:32.867798 21238 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:32.874485 21185 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:32.950623 17726 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.031s sys 0.043s
I20260810 18:52:32.344199 21275 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:32.971841 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:32.973452 17727 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.623s user 0.019s sys 0.042s
I20260810 18:52:32.936144 21142 raft_consensus.cc:3032] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 LEADER]: Stepping down as leader of term 8
I20260810 18:52:32.903532 21277 raft_consensus.cc:504] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:32.936230 21174 leader_election.cc:288] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:32.864662 21268 consensus_queue.cc:237] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:32.911490 21223 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:32.898227 21228 raft_consensus.cc:504] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:32.915633 21192 raft_consensus.cc:504] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:32.914841 21273 raft_consensus.cc:504] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:52:32.897496 21257 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:32.888550 21272 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:33.126617 16695 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.907s user 0.033s sys 0.032s
I20260810 18:52:33.009330 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:33.216531 16447 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.065s user 0.043s sys 0.044s
I20260810 18:52:33.617089 21322 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:33.618158 21322 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:33.619007 21214 raft_consensus.cc:504] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:33.610023 20994 raft_consensus.cc:482] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:33.621425 21214 leader_election.cc:288] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:33.247946 21164 raft_consensus.cc:482] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:33.621951 20994 raft_consensus.cc:504] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:33.622843 21164 raft_consensus.cc:504] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:52:33.624554 20994 leader_election.cc:288] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:33.624873 21164 leader_election.cc:288] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:33.625386 16448 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.736s user 0.012s sys 0.012s
I20260810 18:52:33.625288 20994 raft_consensus.cc:482] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:33.625725 16448 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.736s user 0.012s sys 0.012s
W20260810 18:52:33.626035 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 756685us
I20260810 18:52:33.176746 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:33.627775 16448 leader_election.cc:302] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:33.627981 16636 raft_consensus.cc:2454] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:33.629412 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:33.630016 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:33.630642 16702 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.720s user 0.037s sys 0.063s
W20260810 18:52:33.631116 17726 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.255s user 0.049s sys 0.053s
W20260810 18:52:33.643132 16748 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.269s user 0.000s sys 0.025s
W20260810 18:52:33.643465 16748 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.270s user 0.000s sys 0.025s
I20260810 18:52:33.644146 16748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2056) took 1609 ms. Trace:
I20260810 18:52:33.644795 21321 raft_consensus.cc:2781] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:33.645227 21321 raft_consensus.cc:482] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:33.645583 21321 raft_consensus.cc:3037] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:33.645720 21254 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:33.648105 21174 raft_consensus.cc:482] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:33.648523 21228 leader_election.cc:288] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:33.648896 21174 raft_consensus.cc:504] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:33.650058 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:33.650372 21337 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:33.651291 21174 leader_election.cc:288] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:33.650946 21337 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:33.651816 21174 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:33.652894 21174 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:33.653118 16447 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.465s user 0.055s sys 0.059s
W20260810 18:52:33.615890 16695 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.396s user 0.046s sys 0.063s
W20260810 18:52:33.631018 16702 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.720s user 0.037s sys 0.063s
I20260810 18:52:32.952469 21267 raft_consensus.cc:504] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:33.655766 21329 raft_consensus.cc:482] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:33.656296 21329 raft_consensus.cc:504] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:33.657831 21267 leader_election.cc:288] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:33.658867 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:33.659394 16701 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.575s user 0.003s sys 0.006s
W20260810 18:52:33.659760 16701 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.576s user 0.003s sys 0.006s
I20260810 18:52:33.659703 21238 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:33.174566 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:33.663391 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:33.133272 16752 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 7.
I20260810 18:52:33.665294 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:33.665578 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:33.121830 21310 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:33.670205 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:33.635275 16622 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.719s user 0.031s sys 0.066s
W20260810 18:52:33.670943 16622 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.755s user 0.031s sys 0.066s
I20260810 18:52:33.671343 17783 raft_consensus.cc:1261] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260810 18:52:33.630100 16636 raft_consensus.cc:2454] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:33.634208 21273 leader_election.cc:288] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:33.048907 21271 leader_election.cc:288] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:32.945288 21137 raft_consensus.cc:482] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:33.644496 16748 rpcz_store.cc:276] 0810 18:52:32.034829 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:32.035158 (+ 329us) service_pool.cc:227] Handling call
0810 18:52:32.201818 (+166660us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:32.202090 (+ 272us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:32.202123 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:32.242409 (+ 40286us) log.cc:842] Serialized 26 byte log entry
0810 18:52:33.643799 (+1401390us) raft_consensus.cc:1631] Marking committed up to 7
0810 18:52:33.643840 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:33.643905 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:33.643949 (+ 44us) raft_consensus.cc:1671] finished
0810 18:52:33.643962 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:33.644085 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1269925,"threads_started":1,"wal-append.queue_time_us":1335997}
I20260810 18:52:33.658814 21329 leader_election.cc:288] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:33.263984 21182 leader_election.cc:288] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:33.637370 21142 raft_consensus.cc:729] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:33.637449 17786 raft_consensus.cc:2454] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 7.
I20260810 18:52:33.638991 21185 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:33.639708 21191 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:33.641024 17727 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.291s user 0.021s sys 0.044s
W20260810 18:52:33.783351 16693 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.001s sys 0.009s
W20260810 18:52:33.783699 16693 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.671s user 0.001s sys 0.009s
I20260810 18:52:33.654646 21223 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:33.659135 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:33.799502 16500 raft_consensus.cc:2454] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 5.
I20260810 18:52:33.664264 16501 raft_consensus.cc:3037] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:33.666158 16508 raft_consensus.cc:2454] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 7.
I20260810 18:52:33.665947 16504 raft_consensus.cc:2454] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
I20260810 18:52:32.944608 21176 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:33.630975 16626 raft_consensus.cc:2454] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:33.048542 21162 leader_election.cc:288] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:33.667871 16695 leader_election.cc:302] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:32.948299 21206 leader_election.cc:288] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:33.634835 16747 raft_consensus.cc:2454] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:52:32.977741 20912 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:33.674840 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:33.670953 16633 raft_consensus.cc:2454] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 7.
I20260810 18:52:33.818922 17783 raft_consensus.cc:2454] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
I20260810 18:52:33.819522 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:32.950827 21244 consensus_queue.cc:1035] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.696s
W20260810 18:52:33.610828 21297 log.cc:924] Time spent T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.431s user 0.000s sys 0.000s
W20260810 18:52:33.484315 21030 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.567s user 0.046s sys 0.053s
W20260810 18:52:33.515764 21193 consensus_meta.cc:287] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 1.507s user 0.008s sys 0.004s
I20260810 18:52:33.686884 21267 raft_consensus.cc:482] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:33.705024 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:33.509276 21304 raft_consensus.cc:482] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:33.714483 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:33.259140 21255 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:33.635176 21028 leader_election.cc:288] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:33.676770 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:33.527701 21155 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:33.614007 21154 leader_election.cc:288] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:33.773379 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:33.864184 16748 raft_consensus.cc:2379] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 5: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:33.559636 21163 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:33.837227 21206 raft_consensus.cc:2781] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:33.717094 16701 leader_election.cc:302] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:33.663792 21080 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:33.779954 16745 raft_consensus.cc:2379] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 4: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:33.727972 16632 raft_consensus.cc:1261] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260810 18:52:33.868525 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 151529us
I20260810 18:52:33.670091 21081 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:33.006414 21187 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:32.950798 21246 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:33.851226 21193 raft_consensus.cc:2726] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:33.703168 21083 raft_consensus.cc:482] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:33.626179 20994 raft_consensus.cc:504] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:33.806577 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 177267us
I20260810 18:52:33.918116 16448 leader_election.cc:302] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:33.918671 16750 raft_consensus.cc:1261] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260810 18:52:33.620831 21322 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:33.817705 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 149992us
I20260810 18:52:33.717378 21273 raft_consensus.cc:482] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:33.672627 21290 raft_consensus.cc:482] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:33.828824 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:33.676913 21310 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:33.835734 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:33.838734 16506 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 7.
I20260810 18:52:33.841735 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:33.805744 21192 leader_election.cc:288] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:33.802841 21329 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:33.845592 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 165201us
I20260810 18:52:33.850194 16505 raft_consensus.cc:2379] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 6: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:33.849781 16631 raft_consensus.cc:2454] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:33.852424 16743 raft_consensus.cc:1226] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260810 18:52:33.856125 21272 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:33.857213 17770 raft_consensus.cc:2454] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
I20260810 18:52:33.775406 21137 raft_consensus.cc:504] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:33.840710 21030 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.923s user 0.046s sys 0.054s
I20260810 18:52:33.828101 21182 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:33.820003 21271 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:33.869277 16701 leader_election.cc:302] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:33.846644 21070 leader_election.cc:288] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:33.216678 21296 raft_consensus.cc:504] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:33.873760 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:33.912460 21081 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:33.969861 16450 leader_election.cc:302] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:33.762503 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:52:33.920830 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 276540us
I20260810 18:52:33.971211 16450 leader_election.cc:302] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:33.972544 21081 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:33.932096 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:33.973436 20994 leader_election.cc:288] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:33.976042 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:33.861941 21304 raft_consensus.cc:504] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:33.933230 21154 raft_consensus.cc:2781] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260810 18:52:33.918871 21316 log.cc:924] Time spent T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.078s user 0.000s sys 0.001s
I20260810 18:52:33.938804 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:33.941488 16747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43726 (request call id 1554) took 1038 ms. Trace:
I20260810 18:52:33.857987 21267 raft_consensus.cc:504] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:33.916411 20948 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:33.997788 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:33.998226 20948 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:33.916889 21187 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:34.000350 20948 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:34.000952 21187 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:33.948587 20898 raft_consensus.cc:482] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:34.001770 20898 raft_consensus.cc:504] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:34.002596 21265 consensus_meta.cc:287] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 1.118s user 0.007s sys 0.004s
I20260810 18:52:33.949740 20902 raft_consensus.cc:2781] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:33.952364 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:33.953349 16501 raft_consensus.cc:2454] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 8.
I20260810 18:52:33.955729 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:33.956379 21176 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:33.957423 20912 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:33.920372 21083 raft_consensus.cc:504] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:33.959563 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 90362.6us
I20260810 18:52:34.007587 16631 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 7.
I20260810 18:52:33.968396 21257 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:34.007267 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:33.866977 21206 raft_consensus.cc:482] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:33.976696 16626 raft_consensus.cc:2454] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:33.977360 21238 raft_consensus.cc:2781] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:33.985031 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:33.966230 16748 raft_consensus.cc:2454] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
I20260810 18:52:33.964656 21329 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:33.987401 16505 raft_consensus.cc:1261] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:52:33.987978 16695 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:33.988291 16628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60066 (request call id 287) took 3301 ms. Trace:
I20260810 18:52:33.988890 17783 raft_consensus.cc:2454] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 7.
I20260810 18:52:34.014916 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:33.988938 16747 rpcz_store.cc:276] 0810 18:52:32.902777 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:33.841317 (+938540us) service_pool.cc:227] Handling call
0810 18:52:33.941404 (+100087us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:33.990192 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:33.942561 21291 raft_consensus.cc:482] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:33.943514 21244 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:33.992254 21116 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:33.976050 21201 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:33.997530 20994 raft_consensus.cc:482] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:33.998600 16500 raft_consensus.cc:2454] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:33.999042 21137 leader_election.cc:288] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:33.883616 21337 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:33.948884 21277 leader_election.cc:288] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:34.012385 16504 raft_consensus.cc:2454] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:34.013126 16701 leader_election.cc:302] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:33.940133 21273 raft_consensus.cc:504] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:33.986265 21154 raft_consensus.cc:482] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:33.946457 21290 raft_consensus.cc:504] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:33.976892 21271 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:34.016158 16508 raft_consensus.cc:2454] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:34.016875 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:34.015208 16628 rpcz_store.cc:276] 0810 18:52:30.686412 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:30.686709 (+ 297us) service_pool.cc:227] Handling call
0810 18:52:30.706378 (+ 19669us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:31.065651 (+359273us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:31.065697 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:31.066327 (+ 630us) log.cc:842] Serialized 26 byte log entry
0810 18:52:31.182446 (+116119us) raft_consensus.cc:1631] Marking committed up to 8
0810 18:52:31.290478 (+108032us) log.cc:842] Serialized 14 byte log entry
0810 18:52:31.384675 (+ 94197us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:31.384758 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:33.970720 (+2585962us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b18001726c0. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1aee0718b1 00007f1aee07e661 00007f1ae51d2638 00007f1ae51d5b11 00007f1ae51d5ac9 00007f1ae51d5a59 00007f1ae51d5a21 00007f1ae51d4d1d 00007f1aee436204 00007f1aee436039 00007f1ae51b4dc6
0810 18:52:33.977374 (+ 6654us) raft_consensus.cc:1671] finished
0810 18:52:33.988086 (+ 10712us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:33.988229 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":70960,"prepare.queue_time_us":140907,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":225934,"raft.run_cpu_time_us":472,"raft.run_wall_time_us":29405,"spinlock_wait_cycles":28961664,"thread_start_us":580904,"threads_started":4,"wal-append.queue_time_us":301984}
I20260810 18:52:34.010964 21206 raft_consensus.cc:3037] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:33.984958 20942 leader_election.cc:288] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:33.989970 21255 raft_consensus.cc:2726] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:34.018553 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:34.015642 16632 raft_consensus.cc:2454] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
I20260810 18:52:34.055471 21160 consensus_queue.cc:260] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:34.060483 21160 raft_consensus.cc:3037] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
W20260810 18:52:34.071323 16450 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.14.32.195:40171 timed out after 1.515s (SENT) [suppressed 26 similar messages]
W20260810 18:52:33.987352 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 139583us
I20260810 18:52:34.020941 16695 leader_election.cc:302] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:34.021834 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:34.077442 16747 raft_consensus.cc:2454] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 7.
I20260810 18:52:34.078136 16747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43726 (request call id 1552) took 1190 ms. Trace:
I20260810 18:52:34.078553 16747 rpcz_store.cc:276] 0810 18:52:32.887524 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:34.021549 (+1134025us) service_pool.cc:227] Handling call
0810 18:52:34.078062 (+ 56513us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:34.020675 16636 raft_consensus.cc:2454] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:33.996692 21304 leader_election.cc:288] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:34.003931 21174 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:34.108830 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:34.006484 20902 raft_consensus.cc:482] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:33.848899 21142 consensus_queue.cc:260] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:33.997833 21344 raft_consensus.cc:2726] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260810 18:52:34.032943 21343 log.cc:924] Time spent T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20260810 18:52:34.034102 21329 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:34.036639 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:33.977181 21182 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:34.022464 21291 raft_consensus.cc:504] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:34.022668 21272 raft_consensus.cc:2726] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:34.040838 20994 raft_consensus.cc:504] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:34.042507 21267 leader_election.cc:288] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:34.043023 21277 raft_consensus.cc:2781] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:34.042928 16701 leader_election.cc:302] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:34.044699 16633 raft_consensus.cc:2454] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:34.042536 16740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43726 (request call id 1545) took 2436 ms. Trace:
I20260810 18:52:34.047905 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:34.133689 16505 raft_consensus.cc:2379] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 5: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:34.047909 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:34.048444 21296 leader_election.cc:288] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:34.028970 20898 leader_election.cc:288] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:34.048884 21154 raft_consensus.cc:3037] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:34.049671 21083 leader_election.cc:288] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:34.014568 21238 raft_consensus.cc:482] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:34.052125 16501 raft_consensus.cc:2454] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 7.
I20260810 18:52:34.045325 21273 leader_election.cc:288] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:34.003047 21265 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:34.055702 21328 log.cc:924] Time spent T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.092s user 0.001s sys 0.000s
W20260810 18:52:34.107714 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 86854.4us
I20260810 18:52:34.154704 16695 leader_election.cc:302] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:34.155481 21069 leader_election.cc:288] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:34.155965 16695 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:34.158617 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:34.160254 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 88989.9us
I20260810 18:52:34.110041 17726 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:52:34.109030 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:34.117911 20902 raft_consensus.cc:3037] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:34.119912 21334 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:34.163280 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:34.148052 21329 raft_consensus.cc:2781] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:34.150089 21238 raft_consensus.cc:3037] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:34.123270 17727 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:34.123713 21308 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:34.030241 21244 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:34.134570 16500 raft_consensus.cc:2454] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:34.115186 20942 raft_consensus.cc:2781] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260810 18:52:34.138103 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 220037us
I20260810 18:52:34.186388 16448 leader_election.cc:302] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:34.132819 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 90021.3us
I20260810 18:52:34.133291 16740 rpcz_store.cc:276] 0810 18:52:31.605801 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:31.606003 (+ 202us) service_pool.cc:227] Handling call
0810 18:52:31.606498 (+ 495us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:31.992149 (+385651us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:31.992193 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:31.999746 (+ 7553us) log.cc:842] Serialized 26 byte log entry
0810 18:52:32.207555 (+207809us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:32.207611 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:34.042097 (+1834486us) spinlock_profiling.cc:243] Waited 373 ms on lock 0x7b540007f8d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:34.042161 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:34.042217 (+ 56us) raft_consensus.cc:1671] finished
0810 18:52:34.042231 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:34.042466 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":962,"raft.queue_time_us":5164,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":8323,"spinlock_wait_cycles":858965376,"thread_start_us":40673,"threads_started":1,"wal-append.queue_time_us":711916,"wal-append.run_cpu_time_us":2198,"wal-append.run_wall_time_us":1156853}
I20260810 18:52:34.129869 21142 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:34.151883 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:34.189527 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:34.152053 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:34.190194 16501 raft_consensus.cc:2454] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 7.
I20260810 18:52:34.161499 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:34.140318 21277 raft_consensus.cc:482] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:34.163131 20912 raft_consensus.cc:2726] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260810 18:52:34.180848 21053 raft_consensus.cc:2781] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:34.122159 16747 raft_consensus.cc:2454] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:34.196393 21053 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:34.196748 17787 raft_consensus.cc:1261] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260810 18:52:34.196906 21053 raft_consensus.cc:3037] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:34.197114 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:34.198402 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:34.159442 16502 raft_consensus.cc:2454] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:34.167627 21179 raft_consensus.cc:2781] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:34.163833 21290 leader_election.cc:288] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:34.183446 21030 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:34.199437 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:34.201452 21277 raft_consensus.cc:3037] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:34.163748 21334 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:34.167179 21329 raft_consensus.cc:482] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260810 18:52:34.180706 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 57520.7us
I20260810 18:52:34.181051 21064 raft_consensus.cc:2726] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:34.153831 21246 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:34.187326 16448 leader_election.cc:302] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:34.188215 16701 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:34.229507 16701 leader_election.cc:302] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:34.191115 17783 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 7.
I20260810 18:52:34.190277 16740 raft_consensus.cc:2454] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:34.198670 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:34.199035 16747 raft_consensus.cc:2454] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:34.204862 16450 leader_election.cc:302] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:34.186946 20942 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:34.203099 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:34.179705 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 69809.1us
I20260810 18:52:34.234052 16450 leader_election.cc:302] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:34.206825 16501 raft_consensus.cc:2454] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:34.210084 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:34.211603 17727 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:34.235378 16450 leader_election.cc:302] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:34.220239 20994 leader_election.cc:288] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:34.221529 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:34.232043 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:34.232515 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 45256.2us
W20260810 18:52:34.212697 21353 log.cc:924] Time spent T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260810 18:52:34.232651 21349 raft_consensus.cc:2781] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:34.238337 20912 raft_consensus.cc:2781] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:34.232883 21160 consensus_queue.cc:1035] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260810 18:52:34.238754 21203 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:34.255122 21122 raft_consensus.cc:2726] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:34.239557 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:34.238870 20942 raft_consensus.cc:3037] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:34.256438 16740 raft_consensus.cc:2454] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:34.240110 21272 raft_consensus.cc:2781] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:34.240252 20952 raft_consensus.cc:2781] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:34.240717 21266 raft_consensus.cc:2781] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:34.240856 16701 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:34.203099 21179 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:34.241933 21174 raft_consensus.cc:2781] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:34.265189 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:34.242043 21182 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:34.248163 16448 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:34.253382 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:34.255322 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:34.222757 21329 raft_consensus.cc:3037] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:34.255252 21203 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:34.232692 20898 raft_consensus.cc:2781] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:34.260221 16450 leader_election.cc:302] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:34.260553 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:34.262682 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:34.264099 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:34.962169 21265 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:34.197523 21291 leader_election.cc:288] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:35.561542 16740 raft_consensus.cc:2454] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 7.
I20260810 18:52:34.261461 21272 raft_consensus.cc:482] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:35.563117 21272 raft_consensus.cc:3037] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
W20260810 18:52:35.564033 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 1.30381e+06us
I20260810 18:52:34.249922 17780 raft_consensus.cc:2454] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 8.
W20260810 18:52:34.810447 16447 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.900s user 0.064s sys 0.086s
W20260810 18:52:35.567603 16447 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.657s user 0.094s sys 0.140s
I20260810 18:52:35.567679 17780 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55586 (request call id 437) took 1380 ms. Trace:
I20260810 18:52:35.568255 17780 rpcz_store.cc:276] 0810 18:52:34.187536 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:34.189409 (+ 1873us) service_pool.cc:227] Handling call
0810 18:52:35.567613 (+1378204us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:34.792505 16701 leader_election.cc:302] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:34.444789 21339 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:35.571962 21339 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:35.572636 16447 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:35.573772 16447 leader_election.cc:302] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:34.264428 21349 raft_consensus.cc:482] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:35.574962 21349 raft_consensus.cc:3037] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
W20260810 18:52:34.797371 21321 consensus_meta.cc:287] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 1.151s user 0.012s sys 0.000s
I20260810 18:52:35.576025 21321 raft_consensus.cc:504] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:35.447288 21268 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.288s user 0.006s sys 0.032s
W20260810 18:52:35.579290 21268 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.426s user 0.006s sys 0.036s
I20260810 18:52:35.579198 21265 raft_consensus.cc:2781] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:35.509513 16508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1623) took 1516 ms. Trace:
I20260810 18:52:35.579869 21265 raft_consensus.cc:482] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:35.580437 21265 raft_consensus.cc:3037] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:35.580199 16508 rpcz_store.cc:276] 0810 18:52:33.993592 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:34.055439 (+ 61847us) service_pool.cc:227] Handling call
0810 18:52:34.055861 (+ 422us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:34.107095 (+ 51234us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:34.107130 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:34.107659 (+ 529us) log.cc:842] Serialized 26 byte log entry
0810 18:52:34.160226 (+ 52567us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:34.160266 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:34.160336 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:35.509197 (+1348861us) raft_consensus.cc:1671] finished
0810 18:52:35.509225 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:35.509426 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":51692,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":11190,"thread_start_us":39864,"threads_started":1,"wal-append.queue_time_us":1387121}
I20260810 18:52:35.582273 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:34.272300 20994 raft_consensus.cc:2781] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:35.583865 20994 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:35.584486 20994 raft_consensus.cc:3037] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Advancing to term 8
W20260810 18:52:35.434542 21329 consensus_meta.cc:287] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 1.052s user 0.006s sys 0.005s
I20260810 18:52:34.209676 21030 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:34.380465 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:35.587602 16628 raft_consensus.cc:2454] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
I20260810 18:52:35.588496 16628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 2024) took 1310 ms. Trace:
I20260810 18:52:35.588934 16628 rpcz_store.cc:276] 0810 18:52:34.278423 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:34.280308 (+ 1885us) service_pool.cc:227] Handling call
0810 18:52:35.588425 (+1308117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:35.590374 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:35.596092 16495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:52330 (request call id 853) took 2651 ms. Trace:
I20260810 18:52:35.596606 16495 rpcz_store.cc:276] 0810 18:52:32.944746 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:32.944914 (+ 168us) service_pool.cc:227] Handling call
0810 18:52:35.566882 (+2621968us) spinlock_profiling.cc:243] Waited 996 ms on lock 0x7b54000f7154. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807faf0 00007f1ae8102f2a 00007f1aed03fc6e 00007f1ae6736807 00007f1ae6736792 00007f1ae67366c1 00007f1ae673663c 00007f1ae67358f2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:35.595976 (+ 29094us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2291351296}
I20260810 18:52:34.272588 21174 raft_consensus.cc:482] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:35.598428 16750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43726 (request call id 1547) took 2713 ms. Trace:
I20260810 18:52:34.423715 21290 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
W20260810 18:52:34.790702 20902 consensus_meta.cc:287] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.607s user 0.008s sys 0.000s
I20260810 18:52:35.639783 20902 raft_consensus.cc:504] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:34.626586 21053 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:34.438175 21244 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:35.229672 21277 consensus_meta.cc:287] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 1.003s user 0.002s sys 0.010s
I20260810 18:52:35.646270 21277 raft_consensus.cc:504] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:35.649180 21277 leader_election.cc:288] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:35.650002 21277 raft_consensus.cc:2781] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:35.650897 21277 raft_consensus.cc:482] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:34.257318 20912 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:35.652243 20912 raft_consensus.cc:3037] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Advancing to term 8
W20260810 18:52:35.430446 16695 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.262s user 0.009s sys 0.020s
W20260810 18:52:35.659739 16695 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.492s user 0.011s sys 0.021s
W20260810 18:52:35.660175 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 1.49376e+06us
I20260810 18:52:34.445278 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:35.550711 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:35.664343 16632 raft_consensus.cc:2454] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
W20260810 18:52:35.665024 16632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 2022) took 1387 ms (client timeout 1644 ms). Trace:
W20260810 18:52:35.665452 16632 rpcz_store.cc:269] 0810 18:52:34.277661 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:34.279292 (+ 1631us) service_pool.cc:227] Handling call
0810 18:52:35.664956 (+1385664us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:35.666688 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:35.668470 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:35.669708 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:35.670825 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:35.561550 16500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52352 (request call id 1632) took 1500 ms. Trace:
I20260810 18:52:35.671528 16500 rpcz_store.cc:276] 0810 18:52:34.060632 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:34.250265 (+189633us) service_pool.cc:227] Handling call
0810 18:52:35.561436 (+1311171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:35.672545 16632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60066 (request call id 310) took 1242 ms. Trace:
I20260810 18:52:35.672996 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:35.674238 16500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52352 (request call id 1642) took 1501 ms. Trace:
I20260810 18:52:34.262687 20952 raft_consensus.cc:482] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260810 18:52:35.563022 16740 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43726 (request call id 1559) took 1902 ms (client timeout 2007 ms). Trace:
I20260810 18:52:35.675366 20952 raft_consensus.cc:3037] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
W20260810 18:52:35.675545 16740 rpcz_store.cc:269] 0810 18:52:33.661049 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:34.263848 (+602799us) service_pool.cc:227] Handling call
0810 18:52:35.562950 (+1299102us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 18:52:35.676774 16740 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37716 (request call id 412) took 1833 ms (client timeout 1832 ms). Trace:
W20260810 18:52:35.677129 16740 rpcz_store.cc:269] 0810 18:52:33.847668 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:35.676546 (+1828878us) service_pool.cc:214] Skipping call since client already timed out
0810 18:52:35.676675 (+ 129us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260810 18:52:35.676930 21349 raft_consensus.cc:504] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:35.678027 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:35.678799 16740 raft_consensus.cc:2454] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
W20260810 18:52:35.679916 16740 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43738 (request call id 2067) took 1614 ms (client timeout 1621 ms). Trace:
W20260810 18:52:35.680383 16740 rpcz_store.cc:269] 0810 18:52:34.065127 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:35.677778 (+1612651us) service_pool.cc:227] Handling call
0810 18:52:35.679852 (+ 2074us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:35.681411 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:35.682099 16740 raft_consensus.cc:2454] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
W20260810 18:52:35.682613 16740 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43726 (request call id 1561) took 2021 ms (client timeout 2071 ms). Trace:
I20260810 18:52:35.674619 16500 rpcz_store.cc:276] 0810 18:52:34.172654 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:35.672673 (+1500019us) service_pool.cc:227] Handling call
0810 18:52:35.674161 (+ 1488us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:35.684306 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:35.685788 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:35.682989 16740 rpcz_store.cc:269] 0810 18:52:33.661625 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:35.681103 (+2019478us) service_pool.cc:227] Handling call
0810 18:52:35.682562 (+ 1459us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:35.687230 16500 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d"
dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
mode: GRACEFUL
new_leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
from {username='slave'} at 127.0.0.1:51572
I20260810 18:52:35.687252 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:35.687952 16500 raft_consensus.cc:595] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Received request to transfer leadership to TS 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:52:35.688395 16740 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43738 (request call id 2085) took 1430 ms (client timeout 1515 ms). Trace:
I20260810 18:52:35.688529 21349 leader_election.cc:288] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:35.688678 16500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:51572 (request call id 14) took 1412 ms. Trace:
W20260810 18:52:34.652925 16745 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43726 (request call id 1553) took 1765 ms (client timeout 1717 ms). Trace:
I20260810 18:52:35.689217 21349 consensus_queue.cc:1035] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260810 18:52:35.689424 16745 rpcz_store.cc:269] 0810 18:52:32.887817 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:32.913042 (+ 25225us) service_pool.cc:227] Handling call
0810 18:52:34.652665 (+1739623us) spinlock_profiling.cc:243] Waited 808 ms on lock 0x7b540034eed4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:34.652833 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1857569152}
I20260810 18:52:35.690711 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:35.691304 16745 raft_consensus.cc:2454] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
W20260810 18:52:35.691732 16745 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43738 (request call id 2072) took 1623 ms (client timeout 1713 ms). Trace:
W20260810 18:52:35.692049 16745 rpcz_store.cc:269] 0810 18:52:34.068127 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:35.690450 (+1622323us) service_pool.cc:227] Handling call
0810 18:52:35.691685 (+ 1235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:35.692780 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:35.693503 16745 raft_consensus.cc:2396] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
W20260810 18:52:35.693872 16745 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43738 (request call id 2065) took 1629 ms (client timeout 1741 ms). Trace:
W20260810 18:52:35.694165 16745 rpcz_store.cc:269] 0810 18:52:34.064422 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:35.692577 (+1628155us) service_pool.cc:227] Handling call
0810 18:52:35.693834 (+ 1257us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:34.265974 21266 raft_consensus.cc:482] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:35.695308 21266 raft_consensus.cc:3037] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:35.697065 21349 raft_consensus.cc:2781] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:35.697636 21349 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:35.698092 21349 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Advancing to term 8
W20260810 18:52:35.698542 16740 rpcz_store.cc:269] 0810 18:52:34.258239 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:35.686938 (+1428699us) service_pool.cc:227] Handling call
0810 18:52:35.688270 (+ 1332us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:35.699568 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:35.598734 21174 raft_consensus.cc:3037] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:35.701843 16504 raft_consensus.cc:2379] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 5: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:34.782249 21203 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:35.598850 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:34.281401 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:35.703655 16495 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 7.
W20260810 18:52:35.703879 16747 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43738 (request call id 2077) took 1632 ms (client timeout 1599 ms). Trace:
W20260810 18:52:35.704319 16747 rpcz_store.cc:269] 0810 18:52:34.071425 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:34.281132 (+209707us) service_pool.cc:227] Handling call
0810 18:52:35.703797 (+1422665us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:35.704353 16495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52352 (request call id 1637) took 1642 ms. Trace:
I20260810 18:52:35.704978 16495 rpcz_store.cc:276] 0810 18:52:34.062236 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:35.598521 (+1536285us) service_pool.cc:227] Handling call
0810 18:52:35.704282 (+105761us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:35.705286 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:34.798738 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:35.706676 16495 raft_consensus.cc:1261] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260810 18:52:35.707643 16495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1625) took 1713 ms. Trace:
I20260810 18:52:35.708249 16495 rpcz_store.cc:276] 0810 18:52:33.994246 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:35.705808 (+1711562us) service_pool.cc:227] Handling call
0810 18:52:35.706143 (+ 335us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:35.707346 (+ 1203us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:35.707581 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:35.706449 16502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52352 (request call id 1630) took 1646 ms. Trace:
I20260810 18:52:35.637188 16750 rpcz_store.cc:276] 0810 18:52:32.885142 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:32.886026 (+ 884us) service_pool.cc:227] Handling call
0810 18:52:32.886390 (+ 364us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:33.919117 (+1032727us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:35.598198 (+1679081us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b5400156654. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:35.598368 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2368209792}
I20260810 18:52:35.710243 16495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1628) took 1651 ms. Trace:
I20260810 18:52:35.638221 21290 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:34.192951 21308 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:35.711913 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:34.564482 21206 raft_consensus.cc:504] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:34.279709 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:35.785852 16622 raft_consensus.cc:2454] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:35.786586 16622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 2023) took 1508 ms. Trace:
I20260810 18:52:35.787014 16622 rpcz_store.cc:276] 0810 18:52:34.278087 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:34.279394 (+ 1307us) service_pool.cc:227] Handling call
0810 18:52:35.786512 (+1507118us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:35.586097 21329 raft_consensus.cc:504] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:34.281167 21255 raft_consensus.cc:2781] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260810 18:52:35.796983 16575 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.304s user 0.004s sys 0.012s
W20260810 18:52:35.797537 16575 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.304s user 0.004s sys 0.012s
W20260810 18:52:35.797951 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 1.30498e+06us
W20260810 18:52:35.801564 16750 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37716 (request call id 411) took 1976 ms (client timeout 2172 ms). Trace:
W20260810 18:52:35.802527 16750 rpcz_store.cc:269] 0810 18:52:33.825469 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:35.710528 (+1885059us) service_pool.cc:227] Handling call
0810 18:52:35.801485 (+ 90957us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:35.806118 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:35.702852 20902 leader_election.cc:288] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:35.709661 16502 rpcz_store.cc:276] 0810 18:52:34.060129 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:34.312057 (+251928us) service_pool.cc:227] Handling call
0810 18:52:35.706382 (+1394325us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:35.590809 16506 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56468 (request call id 301) took 1932 ms. Trace:
I20260810 18:52:35.817456 16506 rpcz_store.cc:276] 0810 18:52:33.658419 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:33.658552 (+ 133us) service_pool.cc:227] Handling call
0810 18:52:35.590646 (+1932094us) spinlock_profiling.cc:243] Waited 179 ms on lock 0x7b54000f7154. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:35.590752 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":411404160}
I20260810 18:52:35.714020 21308 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:35.651592 21244 raft_consensus.cc:482] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:35.710728 16495 rpcz_store.cc:276] 0810 18:52:34.059388 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:35.709277 (+1649889us) service_pool.cc:227] Handling call
0810 18:52:35.709659 (+ 382us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:35.709875 (+ 216us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:35.709913 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:35.709970 (+ 57us) raft_consensus.cc:1631] Marking committed up to 7
0810 18:52:35.710016 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:35.710074 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:35.710185 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:35.689220 16500 rpcz_store.cc:276] 0810 18:52:34.276728 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:35.687019 (+1410291us) service_pool.cc:227] Handling call
0810 18:52:35.688613 (+ 1594us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 18:52:35.591315 16628 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 2026) took 1312 ms (client timeout 1616 ms). Trace:
I20260810 18:52:35.733263 21265 raft_consensus.cc:504] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:35.675372 21053 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:34.391692 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:35.784304 16631 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.404s user 0.012s sys 0.014s
W20260810 18:52:35.845489 16631 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.490s user 0.013s sys 0.014s
I20260810 18:52:35.662735 16695 leader_election.cc:302] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:34.265467 21179 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:34.280990 20898 raft_consensus.cc:482] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260810 18:52:34.902300 21154 consensus_meta.cc:287] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.751s user 0.006s sys 0.005s
W20260810 18:52:35.783540 20894 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.580s user 0.006s sys 0.023s
W20260810 18:52:35.821103 17786 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55586 (request call id 428) took 3568 ms (client timeout 1614 ms). Trace:
I20260810 18:52:35.823433 16621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1627) took 4198 ms. Trace:
W20260810 18:52:35.823315 16628 rpcz_store.cc:269] 0810 18:52:34.279056 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:35.590071 (+1311015us) service_pool.cc:227] Handling call
0810 18:52:35.591261 (+ 1190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:35.824440 16747 raft_consensus.cc:2379] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 5: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:35.851172 16621 rpcz_store.cc:276] 0810 18:52:31.618611 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:31.618749 (+ 138us) service_pool.cc:227] Handling call
0810 18:52:31.619057 (+ 308us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:31.928700 (+309643us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:35.815323 (+3886623us) spinlock_profiling.cc:243] Waited 306 ms on lock 0x7b540035dc54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:35.817433 (+ 2110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":704190848}
I20260810 18:52:35.818558 21277 raft_consensus.cc:3037] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
W20260810 18:52:35.850422 17786 rpcz_store.cc:269] 0810 18:52:32.215560 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:32.215733 (+ 173us) service_pool.cc:227] Handling call
0810 18:52:35.784096 (+3568363us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54006a27d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:35.784272 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3108804864}
I20260810 18:52:35.578955 21321 leader_election.cc:288] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:34.804649 21271 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:35.838050 16502 raft_consensus.cc:1226] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:35.861397 16502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1639) took 1798 ms. Trace:
I20260810 18:52:35.861775 16502 rpcz_store.cc:276] 0810 18:52:34.062644 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:35.817234 (+1754590us) service_pool.cc:227] Handling call
0810 18:52:35.837887 (+ 20653us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:35.861117 (+ 23230us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:35.861320 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:35.839103 16506 raft_consensus.cc:1261] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260810 18:52:35.685753 20952 raft_consensus.cc:504] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:35.866554 20952 leader_election.cc:288] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:35.839322 16500 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af"
dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
mode: GRACEFUL
new_leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
from {username='slave'} at 127.0.0.1:51572
W20260810 18:52:35.779778 17724 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.536s user 0.000s sys 0.016s
I20260810 18:52:35.867342 16500 raft_consensus.cc:595] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Received request to transfer leadership to TS 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:52:35.867429 17724 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.624s user 0.000s sys 0.017s
W20260810 18:52:35.868012 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 1.64464e+06us
I20260810 18:52:35.840358 21349 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:35.869819 21349 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:35.792263 21255 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
W20260810 18:52:35.870393 17724 connection.cc:232] RPC call timeout handler was delayed by -1.5454024020582438s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.263s for a response
W20260810 18:52:35.850795 16626 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 2025) took 1572 ms (client timeout 1538 ms). Trace:
W20260810 18:52:35.871490 16626 rpcz_store.cc:269] 0810 18:52:34.278743 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:34.280417 (+ 1674us) service_pool.cc:227] Handling call
0810 18:52:35.850716 (+1570299us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 18:52:35.849277 21238 consensus_meta.cc:287] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 1.663s user 0.013s sys 0.000s
I20260810 18:52:35.694870 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:35.875619 16745 raft_consensus.cc:2454] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
W20260810 18:52:35.876253 16745 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43738 (request call id 2064) took 1812 ms (client timeout 1786 ms). Trace:
I20260810 18:52:35.849541 20898 raft_consensus.cc:3037] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:35.870555 21255 raft_consensus.cc:3037] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:35.835431 20902 raft_consensus.cc:2781] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:35.852528 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:35.883649 16750 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43738 (request call id 2063) took 1820 ms (client timeout 2064 ms). Trace:
W20260810 18:52:35.883980 16750 rpcz_store.cc:269] 0810 18:52:34.063446 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:35.825676 (+1762230us) service_pool.cc:227] Handling call
0810 18:52:35.883586 (+ 57910us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:35.673036 16632 rpcz_store.cc:276] 0810 18:52:34.430684 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:35.671741 (+1241057us) service_pool.cc:227] Handling call
0810 18:52:35.672024 (+ 283us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:35.672190 (+ 166us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:35.672219 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:35.672261 (+ 42us) raft_consensus.cc:1631] Marking committed up to 6
0810 18:52:35.672294 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:35.672344 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:35.672481 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 18:52:35.850337 20894 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.650s user 0.007s sys 0.023s
I20260810 18:52:35.873428 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:35.851862 21142 consensus_meta.cc:287] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 1.651s user 0.007s sys 0.007s
I20260810 18:52:35.818374 21339 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:35.853384 16575 connection.cc:232] RPC call timeout handler was delayed by -1.1096313057933003s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1945s for a response
W20260810 18:52:35.892143 16575 connection.cc:232] RPC call timeout handler was delayed by -1.0546658288221806s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2189s for a response
W20260810 18:52:35.877465 16745 rpcz_store.cc:269] 0810 18:52:34.064046 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:35.694675 (+1630629us) service_pool.cc:227] Handling call
0810 18:52:35.876176 (+181501us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:35.893283 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:35.874881 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:35.836100 21244 raft_consensus.cc:504] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:35.891548 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 1.09906e+06us
I20260810 18:52:35.894676 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:35.903928 16750 raft_consensus.cc:2454] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
W20260810 18:52:35.904458 16750 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43738 (request call id 2075) took 1834 ms (client timeout 2151 ms). Trace:
W20260810 18:52:35.904846 16750 rpcz_store.cc:269] 0810 18:52:34.070207 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:35.885516 (+1815309us) service_pool.cc:227] Handling call
0810 18:52:35.904393 (+ 18877us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:35.895239 16621 raft_consensus.cc:2379] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 5: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
W20260810 18:52:35.896389 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 322687us
I20260810 18:52:35.897234 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:35.899423 16745 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43738 (request call id 2084) took 1696 ms (client timeout 2063 ms). Trace:
W20260810 18:52:35.906858 16745 rpcz_store.cc:269] 0810 18:52:34.203413 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:35.893016 (+1689603us) service_pool.cc:227] Handling call
0810 18:52:35.899350 (+ 6334us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:35.907734 16500 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd"
dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
mode: GRACEFUL
new_leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
from {username='slave'} at 127.0.0.1:51572
I20260810 18:52:35.907975 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:35.908252 16632 raft_consensus.cc:2379] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 6: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:35.881875 20902 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:35.908901 16745 raft_consensus.cc:2396] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260810 18:52:35.855893 21154 raft_consensus.cc:504] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:35.909286 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 342403us
I20260810 18:52:35.908339 16500 raft_consensus.cc:595] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Received request to transfer leadership to TS 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:35.919649 16500 raft_consensus.cc:603] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260810 18:52:35.913968 17726 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.004s sys 0.022s
W20260810 18:52:35.920996 17726 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.511s user 0.004s sys 0.023s
I20260810 18:52:35.894389 21268 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:35.926904 21239 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:35.927371 21268 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:35.928920 21384 consensus_queue.cc:1035] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:52:35.930212 21384 raft_consensus.cc:2781] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:35.930540 21384 raft_consensus.cc:482] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:35.915580 20902 raft_consensus.cc:3037] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:35.931217 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:35.931429 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:35.932008 16495 raft_consensus.cc:2379] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 8: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:35.932310 16628 raft_consensus.cc:2396] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for term 9 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260810 18:52:35.930912 21268 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:35.930913 21384 raft_consensus.cc:3037] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:35.938408 21244 leader_election.cc:288] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:35.930891 21174 raft_consensus.cc:504] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:35.940253 16505 raft_consensus.cc:2379] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 5: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:35.941362 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:35.941931 20979 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:35.942189 16628 raft_consensus.cc:2396] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260810 18:52:35.918011 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:35.942891 17771 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55586 (request call id 426) took 4377 ms. Trace:
W20260810 18:52:35.943506 16750 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43738 (request call id 2070) took 1876 ms (client timeout 2224 ms). Trace:
I20260810 18:52:35.943290 17771 rpcz_store.cc:276] 0810 18:52:31.565804 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:31.566983 (+ 1179us) service_pool.cc:227] Handling call
0810 18:52:31.567414 (+ 431us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:31.594327 (+ 26913us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:31.594388 (+ 61us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:31.594453 (+ 65us) raft_consensus.cc:1631] Marking committed up to 6
0810 18:52:31.594501 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:35.942657 (+4348156us) spinlock_profiling.cc:243] Waited 26.6 ms on lock 0x7b540059f954. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:35.942699 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:35.942812 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":61252864}
W20260810 18:52:35.944000 16750 rpcz_store.cc:269] 0810 18:52:34.066801 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:35.905754 (+1838953us) service_pool.cc:227] Handling call
0810 18:52:35.943448 (+ 37694us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:35.945127 21385 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:35.945420 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:35.945709 21385 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:35.946733 20994 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:35.947723 21265 leader_election.cc:288] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:35.948042 21385 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:35.904019 21290 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:35.949195 16450 leader_election.cc:302] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:35.949678 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:35.950327 16626 raft_consensus.cc:2454] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 8.
I20260810 18:52:35.943221 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:35.951128 16628 raft_consensus.cc:3037] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:35.951872 16743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43726 (request call id 1548) took 3065 ms. Trace:
I20260810 18:52:35.952240 16743 rpcz_store.cc:276] 0810 18:52:32.886274 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:32.888276 (+ 2002us) service_pool.cc:227] Handling call
0810 18:52:32.888654 (+ 378us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:33.947426 (+1058772us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:35.951634 (+2004208us) spinlock_profiling.cc:243] Waited 963 ms on lock 0x7b54003bef54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:35.951813 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2215902080}
I20260810 18:52:35.953181 16504 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52352 (request call id 1624) took 1959 ms. Trace:
I20260810 18:52:35.953297 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:35.950255 21385 raft_consensus.cc:2781] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:35.925494 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:35.953962 20902 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:35.926542 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:35.954938 16632 raft_consensus.cc:3037] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:35.957963 21154 leader_election.cc:288] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:35.948976 20994 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:35.914215 16745 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37716 (request call id 413) took 1657 ms (client timeout 2104 ms). Trace:
I20260810 18:52:35.953547 16504 rpcz_store.cc:276] 0810 18:52:33.993934 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:34.036244 (+ 42310us) service_pool.cc:227] Handling call
0810 18:52:35.953009 (+1916765us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b54001cc0d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:35.953127 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3632116096}
I20260810 18:52:35.926685 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:35.926043 14467 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260810 18:52:35.960491 20898 raft_consensus.cc:504] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:35.961838 16508 raft_consensus.cc:2379] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 5: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
W20260810 18:52:35.961998 20942 consensus_meta.cc:287] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 1.696s user 0.010s sys 0.000s
I20260810 18:52:35.962405 20942 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:35.964407 20898 leader_election.cc:288] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:35.965466 21275 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:35.954389 16621 raft_consensus.cc:2454] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 7.
W20260810 18:52:35.966552 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 59692.7us
I20260810 18:52:35.880012 21238 raft_consensus.cc:504] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:35.968320 16450 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:35.969410 16743 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43738 (request call id 2066) took 1904 ms (client timeout 2441 ms). Trace:
I20260810 18:52:35.906260 21142 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
W20260810 18:52:35.972008 16743 rpcz_store.cc:269] 0810 18:52:34.064784 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:35.953074 (+1888290us) service_pool.cc:227] Handling call
0810 18:52:35.969329 (+ 16255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:35.976958 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:35.977628 17771 raft_consensus.cc:2454] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:35.977789 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:35.960544 21352 log.cc:924] Time spent T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 1.781s user 0.012s sys 0.032s
I20260810 18:52:35.982236 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:35.986600 16621 raft_consensus.cc:3037] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Advancing to term 8
W20260810 18:52:35.959291 16745 rpcz_store.cc:269] 0810 18:52:34.256623 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:35.907713 (+1651090us) service_pool.cc:227] Handling call
0810 18:52:35.914135 (+ 6422us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:35.987756 16740 raft_consensus.cc:2379] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 7: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:35.977808 16450 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:36.004873 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:36.005137 16450 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:35.988651 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:36.006775 16745 raft_consensus.cc:2379] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 8: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:35.966660 21385 raft_consensus.cc:482] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:36.007828 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:36.008579 16748 raft_consensus.cc:2379] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 5: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:35.983387 17726 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:52:36.009253 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:36.012239 16628 raft_consensus.cc:2454] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
I20260810 18:52:35.971405 21239 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:35.965468 20902 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:36.011247 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:36.019294 16631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60066 (request call id 308) took 1779 ms. Trace:
I20260810 18:52:36.015959 17771 raft_consensus.cc:1226] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:35.983566 21138 raft_consensus.cc:2781] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:36.009402 16450 leader_election.cc:302] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:36.012298 17786 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
I20260810 18:52:35.982044 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:36.021248 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:35.978906 21376 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:36.061465 21376 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:36.023200 16632 raft_consensus.cc:2454] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 7.
I20260810 18:52:36.063027 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:36.063787 21376 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:36.064342 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:36.065447 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:36.065815 16505 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52352 (request call id 1640) took 2003 ms (client timeout 2447 ms). Trace:
W20260810 18:52:36.018859 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 356189us
I20260810 18:52:35.981106 20994 consensus_queue.cc:1046] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260810 18:52:36.019645 16631 rpcz_store.cc:276] 0810 18:52:34.236434 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:34.236573 (+ 139us) service_pool.cc:227] Handling call
0810 18:52:34.243059 (+ 6486us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:34.254048 (+ 10989us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:34.254089 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:34.257333 (+ 3244us) log.cc:842] Serialized 26 byte log entry
0810 18:52:35.846344 (+1589011us) raft_consensus.cc:1631] Marking committed up to 8
0810 18:52:36.014886 (+168542us) log.cc:842] Serialized 14 byte log entry
0810 18:52:36.015202 (+ 316us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:36.015268 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:36.015308 (+ 40us) raft_consensus.cc:1671] finished
0810 18:52:36.015321 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:36.015492 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":12977,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":4214,"thread_start_us":1644002,"threads_started":2,"wal-append.queue_time_us":1638361}
I20260810 18:52:36.067224 16695 leader_election.cc:302] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:36.019557 16743 raft_consensus.cc:2454] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
I20260810 18:52:36.067571 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:35.987159 21142 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:36.074399 16695 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:36.074553 16508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52352 (request call id 1641) took 1902 ms. Trace:
I20260810 18:52:36.030525 16575 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:36.015141 21154 raft_consensus.cc:482] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:36.075209 21142 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:36.075618 21154 raft_consensus.cc:504] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:36.075986 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:36.077059 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:36.077602 21154 leader_election.cc:288] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:36.008384 21299 raft_consensus.cc:3037] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:36.078209 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:36.032276 17724 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:36.032301 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:36.033349 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:36.079530 16628 raft_consensus.cc:2454] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 7.
I20260810 18:52:36.080394 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:36.080945 16628 raft_consensus.cc:3037] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:36.081092 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:36.081759 16626 raft_consensus.cc:3037] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 5
I20260810 18:52:36.084129 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:36.084748 16631 raft_consensus.cc:2454] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:36.085558 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:36.086076 21172 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:36.086357 16500 raft_consensus.cc:1226] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 9 LEADER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260810 18:52:36.086477 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:36.086727 21172 raft_consensus.cc:3037] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:36.067631 21089 raft_consensus.cc:2781] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:36.087654 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:36.086972 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:36.036852 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:36.068383 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:36.088500 16504 raft_consensus.cc:2354] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 4. Current term is 5.
I20260810 18:52:36.074975 16508 rpcz_store.cc:276] 0810 18:52:34.172296 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:35.582012 (+1409716us) service_pool.cc:227] Handling call
0810 18:52:36.074343 (+492331us) spinlock_profiling.cc:243] Waited 379 ms on lock 0x7b54001cb1d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:36.074478 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":871172864}
I20260810 18:52:36.089172 16743 raft_consensus.cc:2454] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:52:36.044625 21238 leader_election.cc:288] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:36.048420 16752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37716 (request call id 404) took 3325 ms (client timeout 2630 ms). Trace:
W20260810 18:52:36.091630 16752 rpcz_store.cc:269] 0810 18:52:32.723146 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:32.723304 (+ 158us) service_pool.cc:227] Handling call
0810 18:52:36.048159 (+3324855us) spinlock_profiling.cc:243] Waited 352 ms on lock 0x7b5400140d54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:36.048301 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":809518848}
I20260810 18:52:36.092517 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:36.049939 21174 leader_election.cc:288] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:35.969911 21266 raft_consensus.cc:504] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:36.050242 16746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2050) took 4435 ms. Trace:
I20260810 18:52:36.025949 21138 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260810 18:52:36.051479 16749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43738 (request call id 2047) took 4438 ms (client timeout 1515 ms). Trace:
I20260810 18:52:36.052677 16448 leader_election.cc:302] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:36.017773 21339 raft_consensus.cc:2781] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:36.024868 21386 raft_consensus.cc:482] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:36.054723 21277 raft_consensus.cc:504] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:52:36.056124 21206 leader_election.cc:288] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:36.018163 21243 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:36.018355 21330 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:36.058678 21310 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:36.023154 21239 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:36.016302 21385 raft_consensus.cc:3037] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:36.060575 21329 leader_election.cc:288] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:36.061888 21078 leader_election.cc:288] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:36.066378 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 57051.3us
I20260810 18:52:36.024535 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:36.176573 16745 raft_consensus.cc:2379] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 7: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:36.177505 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:36.178166 16745 raft_consensus.cc:2454] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:36.068362 16632 raft_consensus.cc:2379] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 5: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:36.069254 17727 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:36.069788 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:36.017572 20898 raft_consensus.cc:2781] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:36.050343 20942 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:36.071990 21255 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:36.072978 20994 raft_consensus.cc:3032] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 LEADER]: Stepping down as leader of term 4
W20260810 18:52:36.066179 16505 rpcz_store.cc:269] 0810 18:52:34.062831 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:34.312518 (+249687us) service_pool.cc:227] Handling call
0810 18:52:36.065642 (+1753124us) spinlock_profiling.cc:243] Waited 267 ms on lock 0x7b5400145d54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:36.065759 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":613671168}
I20260810 18:52:36.036479 20912 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:36.086560 16502 raft_consensus.cc:1226] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:36.058336 21272 raft_consensus.cc:504] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:36.067924 16621 raft_consensus.cc:2454] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 8.
I20260810 18:52:36.093474 16746 rpcz_store.cc:276] 0810 18:52:31.614993 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:31.615465 (+ 472us) service_pool.cc:227] Handling call
0810 18:52:31.615791 (+ 326us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:32.244611 (+628820us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:32.244660 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:32.244722 (+ 62us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:32.244770 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:36.049869 (+3805099us) spinlock_profiling.cc:243] Waited 628 ms on lock 0x7b540029c554. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:36.049935 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:36.050112 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1445525760}
I20260810 18:52:36.106748 21386 raft_consensus.cc:504] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:36.090921 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 58723.6us
I20260810 18:52:36.098496 16628 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 8.
W20260810 18:52:36.095005 16749 rpcz_store.cc:269] 0810 18:52:31.613042 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:31.613192 (+ 150us) service_pool.cc:227] Handling call
0810 18:52:36.051121 (+4437929us) spinlock_profiling.cc:243] Waited 216 ms on lock 0x7b54002c47d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:36.051258 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":496119680}
I20260810 18:52:36.090565 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:36.100533 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:36.087713 21089 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:36.100427 16695 leader_election.cc:302] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:36.103477 17787 raft_consensus.cc:2454] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
I20260810 18:52:36.078223 21154 raft_consensus.cc:3037] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
W20260810 18:52:36.106557 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 76081.7us
I20260810 18:52:36.273317 16626 raft_consensus.cc:2454] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:36.107504 16501 raft_consensus.cc:2379] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 6: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:36.090998 21151 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:36.071879 21384 raft_consensus.cc:504] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:36.116533 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:36.116820 21266 leader_election.cc:288] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:36.117302 21170 raft_consensus.cc:482] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:36.311327 21170 raft_consensus.cc:504] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:36.313505 21170 leader_election.cc:288] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:36.118109 21380 raft_consensus.cc:482] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:36.118552 16573 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:36.314179 21380 raft_consensus.cc:504] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:36.314602 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 196099us
I20260810 18:52:36.316165 16573 leader_election.cc:398] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Vote denied by peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791) with higher term. Message: Invalid argument: T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 4. Current term is 5.
I20260810 18:52:36.316212 21380 leader_election.cc:288] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:36.316535 16573 leader_election.cc:401] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260810 18:52:36.316620 20912 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:36.118744 21270 raft_consensus.cc:482] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:36.318820 21272 leader_election.cc:288] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:36.120046 16633 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1649) took 2308 ms (client timeout 1574 ms). Trace:
W20260810 18:52:36.121258 16740 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37716 (request call id 406) took 2336 ms (client timeout 2145 ms). Trace:
I20260810 18:52:36.319384 21172 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:36.319717 21290 consensus_queue.cc:1046] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260810 18:52:36.319693 16633 rpcz_store.cc:269] 0810 18:52:33.811983 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:33.819171 (+ 7188us) service_pool.cc:227] Handling call
0810 18:52:36.119711 (+2300540us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b54003692d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:36.119854 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":515972480}
I20260810 18:52:36.321038 21290 raft_consensus.cc:3032] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:36.321604 21290 raft_consensus.cc:729] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:36.321810 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:36.322474 16633 raft_consensus.cc:3037] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 5
I20260810 18:52:36.099436 21339 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:36.323338 21339 raft_consensus.cc:3037] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:36.319393 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:36.326149 16502 raft_consensus.cc:2379] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 8: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:52:36.094790 21391 raft_consensus.cc:2781] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:36.099295 21275 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
W20260810 18:52:36.156816 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 46137.7us
I20260810 18:52:36.332647 16502 raft_consensus.cc:1261] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260810 18:52:36.101810 21276 raft_consensus.cc:2781] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:36.101043 21138 raft_consensus.cc:3037] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:36.185447 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:36.210835 16507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56468 (request call id 298) took 4658 ms. Trace:
I20260810 18:52:36.340631 16448 leader_election.cc:302] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:36.264899 17724 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:36.358628 16448 leader_election.cc:302] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:36.265407 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 165044us
I20260810 18:52:36.363902 16695 leader_election.cc:302] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:36.266461 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:36.267998 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:36.210692 16624 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 295) took 4670 ms (client timeout 3225 ms). Trace:
W20260810 18:52:36.179894 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 110738us
I20260810 18:52:36.113986 20758 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:36.291857 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:36.111847 21245 raft_consensus.cc:482] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:36.113626 21142 raft_consensus.cc:2781] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:36.293514 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:36.274058 16575 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:36.294716 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:36.294960 16631 raft_consensus.cc:3037] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 5
I20260810 18:52:36.296978 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:36.297219 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:36.378944 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:36.376706 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:36.274746 21089 raft_consensus.cc:3037] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:36.118508 21330 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:36.291440 21253 raft_consensus.cc:2781] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:36.297341 21151 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:36.267793 20994 raft_consensus.cc:729] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:36.114313 21243 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:36.310645 21384 leader_election.cc:288] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:36.319839 16740 rpcz_store.cc:269] 0810 18:52:33.784688 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:35.699322 (+1914634us) service_pool.cc:227] Handling call
0810 18:52:36.120998 (+421676us) spinlock_profiling.cc:243] Waited 287 ms on lock 0x7b5400155254. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:36.121178 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":660596480}
I20260810 18:52:36.099046 21200 raft_consensus.cc:2781] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:36.174606 21202 raft_consensus.cc:482] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:36.327553 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:36.329649 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:36.187717 21062 raft_consensus.cc:2781] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:36.184891 20898 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:36.331851 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:36.335249 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:36.339056 16633 raft_consensus.cc:2454] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:36.351172 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:36.365710 16624 rpcz_store.cc:269] 0810 18:52:31.539998 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:31.553092 (+ 13094us) service_pool.cc:227] Handling call
0810 18:52:36.210452 (+4657360us) spinlock_profiling.cc:243] Waited 307 ms on lock 0x7b54003859d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:36.210570 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":705635200}
I20260810 18:52:36.321372 21172 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 9 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:36.367801 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:36.367843 16632 raft_consensus.cc:2454] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
I20260810 18:52:36.194301 20927 leader_election.cc:288] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:36.383716 16749 raft_consensus.cc:2454] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
W20260810 18:52:36.381464 16636 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1645) took 2675 ms (client timeout 2101 ms). Trace:
I20260810 18:52:36.322688 21290 consensus_queue.cc:260] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:36.318821 21270 raft_consensus.cc:504] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:36.598484 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 61914.8us
I20260810 18:52:36.374357 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:52:36.368448 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 103690us
I20260810 18:52:36.598062 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:52:36.598516 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 234697us
I20260810 18:52:36.601514 17724 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:36.391383 16746 raft_consensus.cc:2379] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 4: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
W20260810 18:52:36.520359 16501 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52352 (request call id 1629) took 2460 ms (client timeout 2124 ms). Trace:
W20260810 18:52:36.603716 16501 rpcz_store.cc:269] 0810 18:52:34.059803 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:34.250204 (+190401us) service_pool.cc:227] Handling call
0810 18:52:36.520036 (+2269832us) spinlock_profiling.cc:243] Waited 546 ms on lock 0x7b54000e10d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:36.520230 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1256408960}
I20260810 18:52:36.606619 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:36.385692 16626 raft_consensus.cc:2454] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:36.365335 16621 raft_consensus.cc:2454] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:36.540351 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:36.511160 16742 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43738 (request call id 2054) took 4477 ms (client timeout 2118 ms). Trace:
I20260810 18:52:36.610461 16743 raft_consensus.cc:2454] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
I20260810 18:52:36.612653 17771 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Advancing to term 8
W20260810 18:52:36.556565 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 157729us
W20260810 18:52:36.613590 16742 rpcz_store.cc:269] 0810 18:52:32.034172 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:32.035030 (+ 858us) service_pool.cc:227] Handling call
0810 18:52:36.510872 (+4475842us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b540029d454. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:36.511046 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":316063488}
I20260810 18:52:36.614372 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:36.376869 16500 raft_consensus.cc:1261] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260810 18:52:36.616313 16743 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 9.
I20260810 18:52:36.538602 16496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56468 (request call id 303) took 2862 ms. Trace:
I20260810 18:52:36.615011 16448 leader_election.cc:302] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:52:36.408479 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 134466us
I20260810 18:52:36.557026 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:36.618202 16575 leader_election.cc:302] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:36.608354 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:36.387328 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:36.533300 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:36.619792 16749 raft_consensus.cc:2454] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:36.620113 16505 raft_consensus.cc:2354] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 4. Current term is 5.
I20260810 18:52:36.620309 17786 raft_consensus.cc:2379] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 7: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:52:36.615034 16495 raft_consensus.cc:2354] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 4. Current term is 5.
I20260810 18:52:36.357084 16507 rpcz_store.cc:276] 0810 18:52:31.552291 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:31.552812 (+ 521us) service_pool.cc:227] Handling call
0810 18:52:31.553166 (+ 354us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:32.340776 (+787610us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:32.340821 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:32.357076 (+ 16255us) log.cc:842] Serialized 26 byte log entry
0810 18:52:32.949248 (+592172us) raft_consensus.cc:1631] Marking committed up to 6
0810 18:52:32.949299 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:36.210005 (+3260706us) spinlock_profiling.cc:243] Waited 361 ms on lock 0x7b5400116554. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:36.210081 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:36.210198 (+ 117us) raft_consensus.cc:1671] finished
0810 18:52:36.210220 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:36.210674 (+ 454us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":51482,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":4409,"spinlock_wait_cycles":829636480,"thread_start_us":687915,"threads_started":2,"wal-append.queue_time_us":1366733,"wal-append.run_cpu_time_us":3220,"wal-append.run_wall_time_us":1317036}
I20260810 18:52:36.597534 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:36.459270 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:36.611233 17787 raft_consensus.cc:2454] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 8.
I20260810 18:52:36.629582 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:36.620744 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:36.602295 16636 rpcz_store.cc:269] 0810 18:52:33.706452 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:33.822420 (+115968us) service_pool.cc:227] Handling call
0810 18:52:36.381225 (+2558805us) spinlock_profiling.cc:243] Waited 83.7 ms on lock 0x7b540035f054. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:36.381381 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":192587520}
I20260810 18:52:36.537096 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:36.621495 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:36.356420 21276 raft_consensus.cc:482] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:36.627754 16508 raft_consensus.cc:1261] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260810 18:52:36.632438 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:36.644464 16740 raft_consensus.cc:2354] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for earlier term 6. Current term is 7.
I20260810 18:52:36.634156 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:36.633317 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:36.644488 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:36.647135 16450 leader_election.cc:398] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Vote denied by peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171) with higher term. Message: Invalid argument: T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for earlier term 6. Current term is 7.
I20260810 18:52:36.647219 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:36.647760 16450 leader_election.cc:401] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260810 18:52:36.628939 16745 raft_consensus.cc:2454] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:52:36.635452 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:36.623934 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:36.635571 16448 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:36.645409 16502 raft_consensus.cc:2454] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 3.
I20260810 18:52:36.652139 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:36.637050 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:36.617709 16496 rpcz_store.cc:276] 0810 18:52:33.659814 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:33.659947 (+ 133us) service_pool.cc:227] Handling call
0810 18:52:33.660239 (+ 292us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:34.003986 (+343747us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:34.004010 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:34.004075 (+ 65us) raft_consensus.cc:1631] Marking committed up to 7
0810 18:52:34.004112 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:36.522411 (+2518299us) spinlock_profiling.cc:243] Waited 344 ms on lock 0x7b54001e14d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:36.522465 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:36.522609 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":790050048}
I20260810 18:52:36.653587 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:36.638720 16636 raft_consensus.cc:1226] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 7. Current term is 8. Ops: []
I20260810 18:52:36.653929 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:36.640663 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:36.652453 16448 leader_election.cc:302] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:36.657191 16448 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:36.658253 16448 leader_election.cc:302] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:36.329994 21391 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:36.664111 16743 raft_consensus.cc:2379] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 5: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:36.664508 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:36.664728 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:52:36.457530 21398 log.cc:924] Time spent T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.086s user 0.000s sys 0.001s
W20260810 18:52:36.431411 21274 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 3.709s user 0.020s sys 0.051s
I20260810 18:52:36.663672 16633 raft_consensus.cc:2396] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260810 18:52:36.440088 21038 raft_consensus.cc:2781] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election won for term 7
I20260810 18:52:36.673070 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:36.673426 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:36.671550 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:36.674595 16745 raft_consensus.cc:2379] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 5: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:36.674443 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:36.666030 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:36.674942 16752 raft_consensus.cc:2454] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 8.
I20260810 18:52:36.667537 16742 raft_consensus.cc:2454] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:36.385321 21142 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:36.456909 21226 raft_consensus.cc:2781] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:36.624756 20898 raft_consensus.cc:3037] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:36.671278 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:36.696947 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:36.699937 16740 raft_consensus.cc:3037] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:36.623293 21062 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:36.673981 16746 raft_consensus.cc:2354] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6.
I20260810 18:52:36.376916 21347 raft_consensus.cc:2781] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260810 18:52:36.681749 21274 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 3.995s user 0.024s sys 0.058s
I20260810 18:52:36.664006 21344 raft_consensus.cc:3037] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:36.379159 20758 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:36.665563 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:36.376971 21186 raft_consensus.cc:2781] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:36.679078 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:36.679785 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:36.669974 16695 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:36.682018 16748 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43726 (request call id 1557) took 3021 ms (client timeout 1586 ms). Trace:
I20260810 18:52:36.746122 16621 raft_consensus.cc:2379] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 5: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
W20260810 18:52:36.682232 16747 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43738 (request call id 2079) took 2609 ms (client timeout 1872 ms). Trace:
I20260810 18:52:36.377142 21133 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:36.383971 21154 raft_consensus.cc:2726] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:36.681587 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:36.671778 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
W20260810 18:52:36.747668 16747 rpcz_store.cc:269] 0810 18:52:34.072652 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:35.705050 (+1632398us) service_pool.cc:227] Handling call
0810 18:52:36.681998 (+976948us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b54002c38d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:36.682151 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":271900544}
I20260810 18:52:36.748356 16743 raft_consensus.cc:2379] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 5: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:36.749033 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:36.376094 21359 raft_consensus.cc:3037] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:36.407874 21245 raft_consensus.cc:504] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:36.378358 21239 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:36.617888 21253 raft_consensus.cc:482] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:36.683421 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:36.680780 16448 leader_election.cc:302] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:36.636191 21202 raft_consensus.cc:504] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:36.488484 21334 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:36.672432 16632 raft_consensus.cc:1226] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260810 18:52:36.717902 16450 leader_election.cc:302] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:36.696655 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:36.643561 21277 leader_election.cc:288] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:36.726706 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:36.789781 16632 raft_consensus.cc:1226] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:36.790289 16500 raft_consensus.cc:3037] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:36.675976 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:36.671250 21176 raft_consensus.cc:3037] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:36.794102 21330 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:36.795341 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:36.796236 17785 raft_consensus.cc:3037] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:36.729624 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:36.356252 21275 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:36.643137 21254 raft_consensus.cc:2726] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:36.734555 17771 raft_consensus.cc:2454] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 8.
I20260810 18:52:36.666694 21342 raft_consensus.cc:2781] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader election won for term 8
I20260810 18:52:36.667914 21276 raft_consensus.cc:3037] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:36.664925 21004 raft_consensus.cc:2726] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260810 18:52:36.680193 21169 raft_consensus.cc:2781] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:52:36.728225 16749 raft_consensus.cc:2454] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
I20260810 18:52:36.676815 21318 raft_consensus.cc:2781] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader election won for term 8
I20260810 18:52:36.684384 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:36.835536 21169 raft_consensus.cc:482] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:36.534739 21386 leader_election.cc:288] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:36.750044 16747 raft_consensus.cc:2379] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 5: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:36.840570 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:36.841691 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:36.842342 16747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37716 (request call id 416) took 1029 ms. Trace:
I20260810 18:52:36.681062 21309 raft_consensus.cc:2781] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:36.751140 16740 raft_consensus.cc:2454] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:36.843964 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:36.749111 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
W20260810 18:52:36.746165 16748 rpcz_store.cc:269] 0810 18:52:33.660448 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:34.108654 (+448206us) service_pool.cc:227] Handling call
0810 18:52:36.681813 (+2573159us) spinlock_profiling.cc:243] Waited 1.84 s on lock 0x7b54002ed6d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:36.681952 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4242218240}
I20260810 18:52:36.844446 21245 leader_election.cc:288] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:36.844754 16743 raft_consensus.cc:3037] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:36.845130 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:36.845826 16748 raft_consensus.cc:2454] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
I20260810 18:52:36.688081 21142 raft_consensus.cc:3037] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:36.847203 20994 consensus_queue.cc:260] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:36.848261 20994 raft_consensus.cc:3037] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:36.751734 21186 raft_consensus.cc:482] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:36.752784 21038 raft_consensus.cc:686] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:36.850167 21186 raft_consensus.cc:3037] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:36.851222 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:36.676311 21210 consensus_queue.cc:1035] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.096s
I20260810 18:52:36.814985 21200 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:36.760918 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:36.761723 21133 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:36.857029 16746 raft_consensus.cc:2379] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 6: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:36.371743 21179 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:36.762212 16742 raft_consensus.cc:3037] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:36.765837 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:36.858420 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
W20260810 18:52:36.771869 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 91173.7us
I20260810 18:52:36.859611 16448 leader_election.cc:302] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:36.729306 21347 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:36.686022 21226 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:36.764616 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 94711.3us
I20260810 18:52:36.686487 21272 consensus_queue.cc:1035] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.082s
I20260810 18:52:36.672323 21391 raft_consensus.cc:3037] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:36.792903 21151 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:36.793031 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:36.752808 21255 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:36.814347 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:36.875370 16750 raft_consensus.cc:2379] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 8: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:36.851531 21342 raft_consensus.cc:686] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 8 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:36.782410 21154 raft_consensus.cc:2781] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:36.765774 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:36.773904 21253 raft_consensus.cc:3037] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:36.784228 21187 raft_consensus.cc:2781] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:36.893345 21187 raft_consensus.cc:482] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:36.893878 21187 raft_consensus.cc:3037] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:36.899361 21243 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:36.901176 21133 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:36.902173 20758 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:36.779592 16631 raft_consensus.cc:2454] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:36.834934 16501 raft_consensus.cc:2454] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:36.837060 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:36.839339 21169 raft_consensus.cc:3037] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:36.721272 21062 raft_consensus.cc:3037] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:36.850672 16749 raft_consensus.cc:1226] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:36.842689 16747 rpcz_store.cc:276] 0810 18:52:35.813423 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:36.841466 (+1028043us) service_pool.cc:227] Handling call
0810 18:52:36.842292 (+ 826us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:36.846657 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:36.760504 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:36.863984 16695 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:36.864722 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:36.873768 16742 raft_consensus.cc:2454] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
I20260810 18:52:36.817106 16626 raft_consensus.cc:3037] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:36.877341 16743 raft_consensus.cc:2454] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:36.879989 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:36.957929 16495 raft_consensus.cc:2454] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:36.861860 21200 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:36.965840 16749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43726 (request call id 1551) took 4078 ms. Trace:
I20260810 18:52:36.966430 16749 rpcz_store.cc:276] 0810 18:52:32.887242 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:36.850223 (+3962981us) service_pool.cc:227] Handling call
0810 18:52:36.850503 (+ 280us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:36.965567 (+115064us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:36.965780 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:36.863205 21347 raft_consensus.cc:3037] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Advancing to term 8
W20260810 18:52:36.914271 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 196439us
I20260810 18:52:36.867015 21226 raft_consensus.cc:3037] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:36.928596 16752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2059) took 3271 ms. Trace:
I20260810 18:52:36.997552 17786 raft_consensus.cc:1261] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260810 18:52:36.997351 16752 rpcz_store.cc:276] 0810 18:52:33.657035 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:36.875507 (+3218472us) service_pool.cc:227] Handling call
0810 18:52:36.927965 (+ 52458us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:36.928191 (+ 226us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:36.928215 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:36.928268 (+ 53us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:36.928311 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:36.928390 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:36.928526 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:36.863992 21318 raft_consensus.cc:686] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [term 8 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
W20260810 18:52:36.980242 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 55955.9us
I20260810 18:52:36.890043 21088 raft_consensus.cc:2781] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:36.885572 21309 raft_consensus.cc:686] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:36.908118 21270 leader_election.cc:288] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:37.000442 16448 leader_election.cc:302] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:37.018745 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:36.926810 21296 raft_consensus.cc:2781] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:36.908592 21154 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:37.030234 16752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2069) took 2964 ms. Trace:
I20260810 18:52:36.994781 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:36.936121 21339 consensus_meta.cc:287] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.612s user 0.006s sys 0.005s
W20260810 18:52:36.978467 21322 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.605s user 0.005s sys 0.019s
I20260810 18:52:36.996603 16749 raft_consensus.cc:1226] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:36.923791 21061 raft_consensus.cc:2726] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791) with higher term. Message: Invalid argument: T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 4. Current term is 5.
I20260810 18:52:36.886934 19587 raft_consensus.cc:482] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:36.850749 21038 consensus_queue.cc:237] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:37.019516 16495 raft_consensus.cc:2454] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 7.
I20260810 18:52:37.031530 16628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1664) took 1002 ms. Trace:
I20260810 18:52:36.994002 21243 raft_consensus.cc:2781] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260810 18:52:37.036602 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 49862.7us
I20260810 18:52:37.059959 16450 leader_election.cc:302] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:37.030833 16752 rpcz_store.cc:276] 0810 18:52:34.066177 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:36.998809 (+2932632us) service_pool.cc:227] Handling call
0810 18:52:37.029628 (+ 30819us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:37.029847 (+ 219us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:37.029878 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:37.029933 (+ 55us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:37.029979 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:37.030043 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:37.030167 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:37.044467 16749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2068) took 2979 ms. Trace:
W20260810 18:52:37.044646 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 180736us
I20260810 18:52:37.061138 16749 rpcz_store.cc:276] 0810 18:52:34.065426 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:36.996001 (+2930575us) service_pool.cc:227] Handling call
0810 18:52:36.996377 (+ 376us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:37.044190 (+ 47813us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:37.044400 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:37.015465 21030 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:37.042577 16748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 408) took 3257 ms. Trace:
I20260810 18:52:37.069320 16748 rpcz_store.cc:276] 0810 18:52:33.785405 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:36.933389 (+3147984us) service_pool.cc:227] Handling call
0810 18:52:37.041694 (+108305us) spinlock_profiling.cc:243] Waited 34.1 ms on lock 0x7b5800270f98. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed0502b0 00007f1aed03b7f9 00007f1aed03b2a2 00007f1ae6721684
0810 18:52:37.041953 (+ 259us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:37.042181 (+ 228us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:37.042215 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:37.042270 (+ 55us) raft_consensus.cc:1631] Marking committed up to 7
0810 18:52:37.042315 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:37.042382 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:37.042509 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":78346112}
W20260810 18:52:37.052181 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 417255us
I20260810 18:52:37.052009 16628 rpcz_store.cc:276] 0810 18:52:36.028875 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:36.688241 (+659366us) service_pool.cc:227] Handling call
0810 18:52:36.743338 (+ 55097us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:36.837894 (+ 94556us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:36.837923 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:36.837962 (+ 39us) raft_consensus.cc:1631] Marking committed up to 6
0810 18:52:36.837991 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:37.031249 (+193258us) spinlock_profiling.cc:243] Waited 94.3 ms on lock 0x7b540035eb54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:37.031307 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:37.031463 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":216903552}
I20260810 18:52:37.053337 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:37.071405 16748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2076) took 3000 ms. Trace:
I20260810 18:52:37.071621 16631 raft_consensus.cc:2454] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 8.
I20260810 18:52:37.071878 16748 rpcz_store.cc:276] 0810 18:52:34.070789 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:37.070567 (+2999778us) service_pool.cc:227] Handling call
0810 18:52:37.070872 (+ 305us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:37.071088 (+ 216us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:37.071112 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:37.071160 (+ 48us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:37.071191 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:37.071236 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:37.071342 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 18:52:37.056393 21299 consensus_meta.cc:287] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.978s user 0.011s sys 0.003s
I20260810 18:52:37.063558 16749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2074) took 2994 ms. Trace:
I20260810 18:52:37.035950 21154 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:36.967175 21334 consensus_queue.cc:1046] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.068s
I20260810 18:52:37.066287 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
W20260810 18:52:37.043022 21322 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.717s user 0.006s sys 0.019s
I20260810 18:52:37.042209 21406 raft_consensus.cc:482] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:37.074458 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:37.074838 21179 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:37.043737 21088 raft_consensus.cc:686] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
W20260810 18:52:37.106201 16750 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37716 (request call id 410) took 3293 ms (client timeout 2636 ms). Trace:
I20260810 18:52:37.075134 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:37.107054 16631 raft_consensus.cc:3037] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 6
W20260810 18:52:37.106750 16750 rpcz_store.cc:269] 0810 18:52:33.812520 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:35.945157 (+2132637us) service_pool.cc:227] Handling call
0810 18:52:37.105944 (+1160787us) spinlock_profiling.cc:243] Waited 929 ms on lock 0x7b54000708d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:37.106113 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2136579840}
I20260810 18:52:37.064723 21275 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:37.077257 21385 consensus_meta.cc:287] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.806s user 0.002s sys 0.012s
I20260810 18:52:37.079301 16749 rpcz_store.cc:276] 0810 18:52:34.069680 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:37.061981 (+2992301us) service_pool.cc:227] Handling call
0810 18:52:37.062963 (+ 982us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:37.063180 (+ 217us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:37.063216 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:37.063267 (+ 51us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:37.063307 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:37.063373 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:37.063498 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1402624}
I20260810 18:52:37.080248 17727 leader_election.cc:302] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:37.082293 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:37.029183 21296 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:37.013664 21271 consensus_queue.cc:1046] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
I20260810 18:52:37.085022 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:37.043638 21254 raft_consensus.cc:2781] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election won for term 5
W20260810 18:52:37.130410 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 493642us
I20260810 18:52:37.088536 16752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2073) took 3019 ms. Trace:
I20260810 18:52:37.133395 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:37.134069 16628 raft_consensus.cc:2454] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
I20260810 18:52:37.133333 16752 rpcz_store.cc:276] 0810 18:52:34.069074 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:37.060991 (+2991917us) service_pool.cc:227] Handling call
0810 18:52:37.087921 (+ 26930us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:37.088138 (+ 217us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:37.088173 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:37.088227 (+ 54us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:37.088271 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:37.088335 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:37.088464 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:37.049528 17785 raft_consensus.cc:2454] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 7.
I20260810 18:52:37.057189 19587 raft_consensus.cc:504] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:37.084089 16626 raft_consensus.cc:2454] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:37.095510 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:37.057489 21243 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:37.076022 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:37.112151 16746 raft_consensus.cc:1226] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:37.113124 16742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2080) took 3040 ms. Trace:
I20260810 18:52:37.116040 16747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 407) took 3331 ms. Trace:
I20260810 18:52:37.111217 16743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 409) took 3325 ms. Trace:
I20260810 18:52:37.154142 16743 rpcz_store.cc:276] 0810 18:52:33.785721 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:36.959854 (+3174133us) service_pool.cc:227] Handling call
0810 18:52:37.110431 (+150577us) spinlock_profiling.cc:243] Waited 8.62 ms on lock 0x7b5800270f98. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed0502b0 00007f1aed03b7f9 00007f1aed03b2a2 00007f1ae6721684
0810 18:52:37.110640 (+ 209us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:37.110810 (+ 170us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:37.110837 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:37.110889 (+ 52us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:37.110930 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:37.111005 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:37.111148 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19828480}
I20260810 18:52:37.154014 16747 rpcz_store.cc:276] 0810 18:52:33.785068 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:36.932323 (+3147255us) service_pool.cc:227] Handling call
0810 18:52:37.115212 (+182889us) spinlock_profiling.cc:243] Waited 36.2 ms on lock 0x7b5800270f98. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed0502b0 00007f1aed03b7f9 00007f1aed03b2a2 00007f1ae6721684
0810 18:52:37.115456 (+ 244us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:37.115671 (+ 215us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:37.115705 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:37.115750 (+ 45us) raft_consensus.cc:1631] Marking committed up to 7
0810 18:52:37.115797 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:37.115861 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:37.115975 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":83223296}
I20260810 18:52:37.042764 21339 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:37.120180 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:37.160991 16501 raft_consensus.cc:2379] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 6: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:37.120180 16749 raft_consensus.cc:1226] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260810 18:52:37.088478 16450 leader_election.cc:302] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:37.162339 16749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2082) took 2990 ms. Trace:
I20260810 18:52:37.162822 16743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 414) took 2381 ms. Trace:
I20260810 18:52:37.162669 16749 rpcz_store.cc:276] 0810 18:52:34.172105 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:37.119633 (+2947528us) service_pool.cc:227] Handling call
0810 18:52:37.119958 (+ 325us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:37.162098 (+ 42140us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:37.162289 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:37.163219 16743 rpcz_store.cc:276] 0810 18:52:34.781931 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:37.162086 (+2380155us) service_pool.cc:227] Handling call
0810 18:52:37.162379 (+ 293us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:37.162542 (+ 163us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:37.162565 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:37.162605 (+ 40us) raft_consensus.cc:1631] Marking committed up to 7
0810 18:52:37.162631 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:37.162679 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:37.162773 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:37.135205 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:37.136040 17724 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:37.100126 21406 raft_consensus.cc:504] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:37.166251 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:37.084470 21299 raft_consensus.cc:2726] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:37.094350 16448 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:37.153802 16742 rpcz_store.cc:276] 0810 18:52:34.073257 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:37.112211 (+3038954us) service_pool.cc:227] Handling call
0810 18:52:37.112566 (+ 355us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:37.112760 (+ 194us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:37.112794 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:37.112846 (+ 52us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:37.112884 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:37.112949 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:37.113060 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:37.156829 16747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 415) took 1597 ms. Trace:
I20260810 18:52:37.131166 21296 raft_consensus.cc:3037] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:37.166528 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:37.181589 17783 raft_consensus.cc:2379] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 8: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
W20260810 18:52:37.181857 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 87603.8us
I20260810 18:52:37.173465 16500 raft_consensus.cc:2454] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:37.166843 16752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2083) took 2964 ms. Trace:
I20260810 18:52:37.168944 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:37.184135 16505 raft_consensus.cc:2454] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
I20260810 18:52:37.185065 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:37.171023 16631 raft_consensus.cc:2454] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
I20260810 18:52:37.091276 16695 leader_election.cc:302] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:37.171319 16749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 417) took 1234 ms. Trace:
I20260810 18:52:37.187037 16749 rpcz_store.cc:276] 0810 18:52:35.936821 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:37.170549 (+1233728us) service_pool.cc:227] Handling call
0810 18:52:37.170825 (+ 276us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:37.171017 (+ 192us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:37.171041 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:37.171102 (+ 61us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:37.171131 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:37.171181 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:37.171270 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:37.160328 21243 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Advancing to term 9
W20260810 18:52:37.174497 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 86101.6us
I20260810 18:52:37.177004 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:37.196949 16750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 422) took 1232 ms. Trace:
I20260810 18:52:37.197606 16628 raft_consensus.cc:2396] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260810 18:52:37.178390 16747 rpcz_store.cc:276] 0810 18:52:35.559386 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:37.155836 (+1596450us) service_pool.cc:227] Handling call
0810 18:52:37.156218 (+ 382us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:37.156437 (+ 219us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:37.156474 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:37.156526 (+ 52us) raft_consensus.cc:1631] Marking committed up to 7
0810 18:52:37.156569 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:37.156636 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:37.156764 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:37.197434 16750 rpcz_store.cc:276] 0810 18:52:35.964308 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:37.173415 (+1209107us) service_pool.cc:227] Handling call
0810 18:52:37.196226 (+ 22811us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:37.196550 (+ 324us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:37.196586 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:37.196642 (+ 56us) raft_consensus.cc:1631] Marking committed up to 8
0810 18:52:37.196685 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:37.196749 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:37.196881 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:37.183265 16448 leader_election.cc:302] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:52:37.187657 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 107485us
I20260810 18:52:37.185851 16752 rpcz_store.cc:276] 0810 18:52:34.202532 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:37.136034 (+2933502us) service_pool.cc:227] Handling call
0810 18:52:37.166419 (+ 30385us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:37.166576 (+ 157us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:37.166598 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:37.166634 (+ 36us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:37.166663 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:37.166709 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:37.166794 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:37.199613 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:37.173049 16743 raft_consensus.cc:1226] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:37.200215 16631 raft_consensus.cc:3037] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:37.200522 16743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 421) took 1236 ms. Trace:
I20260810 18:52:37.143247 21202 leader_election.cc:288] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:37.179670 21133 consensus_queue.cc:1035] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.012s
I20260810 18:52:37.189559 16507 raft_consensus.cc:2454] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 7.
I20260810 18:52:37.192880 16495 raft_consensus.cc:3037] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:37.157944 21254 raft_consensus.cc:686] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:37.200870 16743 rpcz_store.cc:276] 0810 18:52:35.963933 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:37.172468 (+1208535us) service_pool.cc:227] Handling call
0810 18:52:37.172805 (+ 337us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:37.200217 (+ 27412us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:37.200468 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:37.204308 16743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2099) took 1018 ms. Trace:
I20260810 18:52:37.202672 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:37.191052 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 99857.6us
I20260810 18:52:37.205484 16450 leader_election.cc:302] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:37.204968 16743 rpcz_store.cc:276] 0810 18:52:36.185991 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:37.203431 (+1017440us) service_pool.cc:227] Handling call
0810 18:52:37.203739 (+ 308us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:37.203971 (+ 232us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:37.204006 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:37.204060 (+ 54us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:37.204091 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:37.204135 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:37.204245 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:37.215021 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:37.215019 21406 leader_election.cc:288] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:37.130528 21385 raft_consensus.cc:504] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:37.082935 21342 consensus_queue.cc:237] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:37.219599 16695 leader_election.cc:302] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:37.170243 21308 raft_consensus.cc:2726] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260810 18:52:37.187597 21188 raft_consensus.cc:2781] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:37.193598 21271 raft_consensus.cc:2781] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:37.222843 21255 consensus_queue.cc:1046] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20260810 18:52:37.240347 19587 leader_election.cc:288] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:37.280055 16626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 2064) took 1158 ms. Trace:
I20260810 18:52:37.280553 16626 rpcz_store.cc:276] 0810 18:52:36.121863 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:36.636765 (+514902us) service_pool.cc:227] Handling call
0810 18:52:37.279806 (+643041us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b54001ebc54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:37.279961 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":376873728}
I20260810 18:52:37.119426 21309 consensus_queue.cc:237] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:37.278218 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 72824.7us
I20260810 18:52:37.111310 21318 consensus_queue.cc:237] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:37.187490 21349 raft_consensus.cc:2726] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:37.343475 21057 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:37.332674 16746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43726 (request call id 1555) took 4379 ms. Trace:
I20260810 18:52:37.332706 21290 raft_consensus.cc:3037] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:37.235954 16752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2093) took 1053 ms. Trace:
I20260810 18:52:37.349690 16746 rpcz_store.cc:276] 0810 18:52:32.903089 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:36.861263 (+3958174us) service_pool.cc:227] Handling call
0810 18:52:37.111948 (+250685us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:37.153747 (+ 41799us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:37.282574 (+128827us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b540004b0d0. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:37.282753 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":541030144}
I20260810 18:52:37.350096 16752 rpcz_store.cc:276] 0810 18:52:36.182143 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:37.200033 (+1017890us) service_pool.cc:227] Handling call
0810 18:52:37.235471 (+ 35438us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:37.235651 (+ 180us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:37.235674 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:37.235713 (+ 39us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:37.235745 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:37.235794 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:37.235898 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:37.359354 21188 raft_consensus.cc:686] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:37.374639 21255 raft_consensus.cc:3032] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:52:37.375137 21255 raft_consensus.cc:729] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:37.376053 21255 consensus_queue.cc:260] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:52:37.378630 16746 raft_consensus.cc:1226] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 7. Current term is 8. Ops: []
I20260810 18:52:37.278772 21285 raft_consensus.cc:2781] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader election won for term 7
I20260810 18:52:37.313845 21320 raft_consensus.cc:2726] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260810 18:52:37.486220 16636 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.740s user 0.035s sys 0.048s
W20260810 18:52:37.486753 16636 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.741s user 0.035s sys 0.048s
I20260810 18:52:37.344451 19587 raft_consensus.cc:2781] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:37.348886 21057 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:37.279965 21271 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:37.208132 21088 consensus_queue.cc:237] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:37.535404 16495 raft_consensus.cc:2454] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
W20260810 18:52:37.537631 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 354444us
I20260810 18:52:37.538674 16448 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:37.539847 16448 leader_election.cc:302] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:52:37.543002 16621 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.719s user 0.026s sys 0.038s
W20260810 18:52:37.543968 16621 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.720s user 0.026s sys 0.038s
I20260810 18:52:37.506359 21271 raft_consensus.cc:3037] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:37.492023 21285 raft_consensus.cc:686] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:37.398053 21391 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:37.385049 21253 raft_consensus.cc:504] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:37.509479 19587 raft_consensus.cc:686] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 6 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
W20260810 18:52:37.568782 16624 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.776s user 0.013s sys 0.056s
W20260810 18:52:37.582562 16624 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.790s user 0.013s sys 0.056s
W20260810 18:52:37.579449 16575 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.960s user 0.053s sys 0.086s
W20260810 18:52:37.580693 16573 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.042s user 0.019s sys 0.079s
W20260810 18:52:37.570019 16623 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.187s user 0.014s sys 0.057s
W20260810 18:52:37.589316 16575 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.970s user 0.054s sys 0.087s
W20260810 18:52:37.589325 16623 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.207s user 0.015s sys 0.058s
W20260810 18:52:37.590390 16573 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.052s user 0.020s sys 0.080s
W20260810 18:52:37.561617 21274 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.750s user 0.041s sys 0.027s
I20260810 18:52:37.595683 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:37.597041 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:37.491595 21254 consensus_queue.cc:237] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:37.604606 16745 raft_consensus.cc:2379] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 5: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:37.603227 17771 raft_consensus.cc:2454] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
I20260810 18:52:37.591789 21339 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:37.594264 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 54512.4us
W20260810 18:52:37.589993 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 971803us
I20260810 18:52:37.576069 21255 raft_consensus.cc:3037] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:37.601007 21411 raft_consensus.cc:482] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:37.598198 16573 leader_election.cc:302] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:52:37.633270 16632 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.758s user 0.028s sys 0.070s
W20260810 18:52:37.655486 16632 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.781s user 0.028s sys 0.070s
I20260810 18:52:37.628715 16575 leader_election.cc:302] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:37.657649 16575 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:37.605044 21299 raft_consensus.cc:2781] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:37.632592 21078 raft_consensus.cc:979] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea: : Instructing follower 980f343dd1fd4e049143f8b9d9f2f1be to start an election
W20260810 18:52:37.655229 21170 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.751s user 0.024s sys 0.057s
I20260810 18:52:37.591424 21380 raft_consensus.cc:2726] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:37.564486 21202 raft_consensus.cc:2781] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:37.575438 21188 consensus_queue.cc:237] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:37.668982 17787 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.566s user 0.010s sys 0.009s
W20260810 18:52:37.684537 17787 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.582s user 0.010s sys 0.009s
I20260810 18:52:37.626217 21385 leader_election.cc:288] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:37.668010 21176 consensus_meta.cc:287] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.874s user 0.006s sys 0.011s
W20260810 18:52:37.603415 21274 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.815s user 0.041s sys 0.027s
W20260810 18:52:37.675274 21170 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.784s user 0.024s sys 0.057s
I20260810 18:52:37.647538 16631 raft_consensus.cc:2454] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
W20260810 18:52:37.669752 21138 consensus_meta.cc:287] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 1.290s user 0.017s sys 0.003s
W20260810 18:52:37.688340 21187 consensus_meta.cc:287] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.794s user 0.011s sys 0.007s
I20260810 18:52:37.679680 21078 raft_consensus.cc:1067] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Signalling peer 980f343dd1fd4e049143f8b9d9f2f1be to start an election
I20260810 18:52:37.689261 21155 raft_consensus.cc:2726] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260810 18:52:37.691740 16633 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.968s user 0.050s sys 0.102s
W20260810 18:52:37.697839 16633 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.974s user 0.050s sys 0.102s
I20260810 18:52:37.681746 21299 raft_consensus.cc:686] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [term 6 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:37.684680 21202 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:37.696122 21187 raft_consensus.cc:504] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:37.692966 21176 raft_consensus.cc:2726] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:37.663196 21411 raft_consensus.cc:504] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:37.666144 21285 consensus_queue.cc:237] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:37.706794 21202 raft_consensus.cc:3037] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:37.729873 21187 leader_election.cc:288] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:37.731796 21411 leader_election.cc:288] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:37.732393 21411 raft_consensus.cc:2781] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:37.732717 21411 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:37.733057 21411 raft_consensus.cc:3037] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:37.695160 21138 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:37.727351 21253 leader_election.cc:288] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:37.745342 21089 consensus_meta.cc:287] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 1.112s user 0.027s sys 0.009s
I20260810 18:52:37.745997 21089 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:37.704483 21057 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:37.752336 21169 consensus_meta.cc:287] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.772s user 0.017s sys 0.000s
I20260810 18:52:37.753017 21169 raft_consensus.cc:504] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:37.735587 21226 consensus_meta.cc:287] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.717s user 0.015s sys 0.002s
W20260810 18:52:37.742375 21243 consensus_meta.cc:287] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.521s user 0.008s sys 0.001s
I20260810 18:52:37.725656 21391 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:37.747635 21347 consensus_meta.cc:287] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.727s user 0.010s sys 0.000s
I20260810 18:52:37.706954 21170 consensus_queue.cc:1035] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260810 18:52:37.741796 21412 raft_consensus.cc:2726] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260810 18:52:37.739620 21186 consensus_meta.cc:287] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.888s user 0.006s sys 0.016s
W20260810 18:52:37.759196 21142 consensus_meta.cc:287] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.912s user 0.014s sys 0.017s
W20260810 18:52:37.773394 21344 consensus_meta.cc:287] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 1.020s user 0.007s sys 0.016s
I20260810 18:52:37.764232 21380 raft_consensus.cc:2781] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260810 18:52:37.763218 21276 consensus_meta.cc:287] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.916s user 0.007s sys 0.014s
I20260810 18:52:37.776252 21142 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:37.777865 21380 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:37.778716 21380 raft_consensus.cc:3037] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:37.778970 21344 raft_consensus.cc:2726] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171) with higher term. Message: Invalid argument: T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for earlier term 6. Current term is 7.
I20260810 18:52:37.778335 21276 raft_consensus.cc:504] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:37.786406 21276 leader_election.cc:288] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:37.768092 21243 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:37.759312 21299 consensus_queue.cc:237] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:37.804399 21334 raft_consensus.cc:3032] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 6 LEADER]: Stepping down as leader of term 6
I20260810 18:52:37.790045 21411 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:37.767755 21347 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:37.885910 16496 raft_consensus.cc:2396] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260810 18:52:37.886230 16495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52352 (request call id 1662) took 1208 ms. Trace:
I20260810 18:52:37.887044 16495 rpcz_store.cc:276] 0810 18:52:36.677669 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:37.145913 (+468244us) service_pool.cc:227] Handling call
0810 18:52:37.885282 (+739369us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b54000f6754. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:37.885807 (+ 525us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26930432}
I20260810 18:52:37.887706 16496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52338 (request call id 1592) took 1102 ms. Trace:
W20260810 18:52:37.888514 16447 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.720s user 0.010s sys 0.005s
W20260810 18:52:37.889207 16447 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.720s user 0.010s sys 0.006s
I20260810 18:52:37.888675 16496 rpcz_store.cc:276] 0810 18:52:36.785359 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:36.858123 (+ 72764us) service_pool.cc:227] Handling call
0810 18:52:37.887390 (+1029267us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54000f5d54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:37.887650 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2357885824}
W20260810 18:52:37.889856 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 721300us
W20260810 18:52:37.894110 17724 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.705s user 0.013s sys 0.001s
W20260810 18:52:37.894615 17724 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.706s user 0.013s sys 0.001s
W20260810 18:52:37.895159 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 712242us
I20260810 18:52:37.891175 16496 raft_consensus.cc:1226] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260810 18:52:37.789113 21142 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:37.769572 21200 consensus_meta.cc:287] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.755s user 0.016s sys 0.001s
I20260810 18:52:37.758113 21226 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:37.925444 16742 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.715s user 0.006s sys 0.010s
I20260810 18:52:37.940687 16502 raft_consensus.cc:2379] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 7: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:37.945441 17775 raft_consensus.cc:2379] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 8: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
W20260810 18:52:37.946048 17727 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.746s user 0.001s sys 0.016s
I20260810 18:52:37.669234 19587 consensus_queue.cc:237] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:37.708873 21004 raft_consensus.cc:3032] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Stepping down as leader of term 7
I20260810 18:52:37.954161 16496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1651) took 1282 ms. Trace:
I20260810 18:52:38.057097 16496 rpcz_store.cc:276] 0810 18:52:36.655031 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:37.889967 (+1234936us) service_pool.cc:227] Handling call
0810 18:52:37.890676 (+ 709us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:37.927787 (+ 37111us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:37.937573 (+ 9786us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 18:52:37.940817 21060 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.748s user 0.011s sys 0.030s
I20260810 18:52:37.810730 21334 raft_consensus.cc:729] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
W20260810 18:52:38.014441 16500 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.008s sys 0.009s
I20260810 18:52:38.027375 21169 leader_election.cc:288] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:38.030150 16748 raft_consensus.cc:1226] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 8. Current term is 9. Ops: []
I20260810 18:52:38.023952 14467 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260810 18:52:38.069691 21347 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:38.071671 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 393006us
I20260810 18:52:38.071782 21004 raft_consensus.cc:729] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:38.072383 21411 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:38.038920 16508 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52352 (request call id 1656) took 1377 ms (client timeout 1787 ms). Trace:
I20260810 18:52:38.034529 21243 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 9 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:38.021486 16450 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.628s user 0.019s sys 0.028s
I20260810 18:52:38.034972 16623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60066 (request call id 289) took 7203 ms. Trace:
W20260810 18:52:38.043668 20898 consensus_meta.cc:287] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 1.289s user 0.012s sys 0.008s
I20260810 18:52:38.077754 20898 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:38.078528 16450 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.686s user 0.020s sys 0.028s
W20260810 18:52:38.079053 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 696081us
I20260810 18:52:38.080890 20898 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:38.080789 16506 raft_consensus.cc:2379] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 8: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
W20260810 18:52:38.073971 21060 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.882s user 0.011s sys 0.030s
I20260810 18:52:38.064693 17783 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55570 (request call id 524) took 2043 ms. Trace:
I20260810 18:52:37.795830 21186 raft_consensus.cc:504] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:38.082062 17783 rpcz_store.cc:276] 0810 18:52:36.020900 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:36.021031 (+ 131us) service_pool.cc:227] Handling call
0810 18:52:38.043607 (+2022576us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b54006a04d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:38.064419 (+ 20812us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2660624128}
I20260810 18:52:38.065382 16621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1665) took 2036 ms. Trace:
I20260810 18:52:37.883008 16342 catalog_manager.cc:5495] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
W20260810 18:52:38.061789 17727 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.862s user 0.002s sys 0.016s
W20260810 18:52:38.074519 16742 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.864s user 0.007s sys 0.011s
W20260810 18:52:38.074685 16508 rpcz_store.cc:269] 0810 18:52:36.661078 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:36.665639 (+ 4561us) service_pool.cc:227] Handling call
0810 18:52:38.038470 (+1372831us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b5400285854. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:38.038686 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3128559488}
I20260810 18:52:38.083945 16621 rpcz_store.cc:276] 0810 18:52:36.029078 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:36.749094 (+720016us) service_pool.cc:227] Handling call
0810 18:52:36.772557 (+ 23463us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:36.772810 (+ 253us) raft_consensus.cc:1520] Early marking committed up to index 5
0810 18:52:36.772834 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:36.784655 (+ 11821us) log.cc:842] Serialized 26 byte log entry
0810 18:52:37.544459 (+759804us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:37.544513 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:37.544604 (+ 91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:38.008852 (+464248us) raft_consensus.cc:1671] finished
0810 18:52:38.016213 (+ 7361us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:38.065138 (+ 48925us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":719939,"threads_started":1,"wal-append.queue_time_us":927455}
I20260810 18:52:38.096873 16508 raft_consensus.cc:1226] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:38.097823 16508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1654) took 1437 ms. Trace:
I20260810 18:52:38.098251 16508 rpcz_store.cc:276] 0810 18:52:36.660223 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:38.095994 (+1435771us) service_pool.cc:227] Handling call
0810 18:52:38.096583 (+ 589us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:38.097465 (+ 882us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:38.097730 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:38.084537 16623 rpcz_store.cc:276] 0810 18:52:30.831324 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:30.831464 (+ 140us) service_pool.cc:227] Handling call
0810 18:52:30.873852 (+ 42388us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:31.386242 (+512390us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:31.386297 (+ 55us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:36.381780 (+4995483us) spinlock_profiling.cc:243] Waited 2.46 ms on lock 0x7b2c000002e8. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae6adde11 00007f1ae6aed442 00007f1ae8135c11 00007f1ae80fa58a 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772
0810 18:52:36.382309 (+ 529us) log.cc:842] Serialized 26 byte log entry
0810 18:52:37.590113 (+1207804us) raft_consensus.cc:1631] Marking committed up to 6
0810 18:52:37.590150 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:38.033900 (+443750us) spinlock_profiling.cc:243] Waited 504 ms on lock 0x7b5400385ed4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:38.034031 (+ 131us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:38.034166 (+ 135us) raft_consensus.cc:1671] finished
0810 18:52:38.034193 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:38.034583 (+ 390us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":64963,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":72059,"spinlock_wait_cycles":1164502272,"thread_start_us":1207308,"threads_started":1,"wal-append.queue_time_us":1346116}
I20260810 18:52:38.099486 21200 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:38.101770 17786 raft_consensus.cc:2354] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 8. Current term is 9.
I20260810 18:52:38.097306 16342 catalog_manager.cc:5495] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 4 to 5, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196). New cstate: current_term: 5 leader_uuid: "831925dd8b644ebab7eb0d672a94eddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260810 18:52:38.103627 16500 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.727s user 0.009s sys 0.009s
I20260810 18:52:38.101651 21138 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:38.109720 16624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1666) took 2032 ms. Trace:
W20260810 18:52:38.109795 16507 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52352 (request call id 1652) took 1450 ms (client timeout 1614 ms). Trace:
W20260810 18:52:38.110508 16507 rpcz_store.cc:269] 0810 18:52:36.659498 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:36.664242 (+ 4744us) service_pool.cc:227] Handling call
0810 18:52:38.103368 (+1439126us) spinlock_profiling.cc:243] Waited 524 ms on lock 0x7b54001392d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:38.109591 (+ 6223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1205844736}
I20260810 18:52:38.109747 21089 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:38.113235 16507 raft_consensus.cc:1226] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260810 18:52:38.113461 16496 raft_consensus.cc:1226] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:38.114084 16507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1589) took 1375 ms. Trace:
I20260810 18:52:38.114367 16496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1586) took 1411 ms. Trace:
I20260810 18:52:38.110422 16624 rpcz_store.cc:276] 0810 18:52:36.029238 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:36.760303 (+731065us) service_pool.cc:227] Handling call
0810 18:52:36.760706 (+ 403us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:36.761144 (+ 438us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:36.761176 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:36.771422 (+ 10246us) log.cc:842] Serialized 26 byte log entry
0810 18:52:37.583113 (+811691us) raft_consensus.cc:1631] Marking committed up to 1
0810 18:52:37.583168 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:37.583254 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:38.008346 (+425092us) raft_consensus.cc:1671] finished
0810 18:52:38.012939 (+ 4593us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:38.061383 (+ 48444us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":334,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":19585,"thread_start_us":790585,"threads_started":1,"wal-append.queue_time_us":939377}
I20260810 18:52:38.118203 16633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1662) took 2089 ms. Trace:
I20260810 18:52:38.119820 16750 raft_consensus.cc:1226] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260810 18:52:38.091958 21004 consensus_queue.cc:260] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:38.129088 21004 raft_consensus.cc:3037] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:38.128989 21334 consensus_queue.cc:260] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:38.129861 16496 rpcz_store.cc:276] 0810 18:52:36.703508 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:38.110758 (+1407250us) service_pool.cc:227] Handling call
0810 18:52:38.113058 (+ 2300us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:38.114038 (+ 980us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:38.114303 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:38.132563 21186 leader_election.cc:288] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:38.135331 16502 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52352 (request call id 1647) took 1484 ms (client timeout 1574 ms). Trace:
W20260810 18:52:38.135815 16502 rpcz_store.cc:269] 0810 18:52:36.651380 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:36.651893 (+ 513us) service_pool.cc:227] Handling call
0810 18:52:38.135034 (+1483141us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54000e1ad4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:38.135260 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2922036096}
W20260810 18:52:38.136706 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 538413us
W20260810 18:52:38.144214 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 64079.2us
I20260810 18:52:38.118796 16633 rpcz_store.cc:276] 0810 18:52:36.028430 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:36.680835 (+652405us) service_pool.cc:227] Handling call
0810 18:52:36.681133 (+ 298us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:36.681360 (+ 227us) raft_consensus.cc:1520] Early marking committed up to index 4
0810 18:52:36.723306 (+ 41946us) log.cc:842] Serialized 14 byte log entry
0810 18:52:38.117193 (+1393887us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:38.117341 (+ 148us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:38.117502 (+ 161us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:38.117670 (+ 168us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:38.117911 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2521,"prepare.queue_time_us":41705,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":326976,"raft.run_wall_time_us":1107,"spinlock_wait_cycles":58909056,"thread_start_us":1298584,"threads_started":2}
I20260810 18:52:38.125733 21380 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:38.114564 16507 rpcz_store.cc:276] 0810 18:52:36.738551 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:38.112004 (+1373453us) service_pool.cc:227] Handling call
0810 18:52:38.112736 (+ 732us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:38.113733 (+ 997us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:38.113972 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:38.135380 16636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1654) took 2051 ms. Trace:
I20260810 18:52:38.154937 21169 consensus_queue.cc:1035] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260810 18:52:38.155303 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:38.155908 21038 consensus_queue.cc:1046] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:52:38.157627 21038 raft_consensus.cc:3032] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Stepping down as leader of term 7
I20260810 18:52:38.148496 21334 raft_consensus.cc:3037] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
W20260810 18:52:38.151190 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 944525us
W20260810 18:52:38.158907 21296 consensus_meta.cc:287] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.954s user 0.008s sys 0.008s
W20260810 18:52:38.193758 17775 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55586 (request call id 440) took 1444 ms (client timeout 1614 ms). Trace:
I20260810 18:52:38.194860 16748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2078) took 4122 ms. Trace:
I20260810 18:52:38.195425 16748 rpcz_store.cc:276] 0810 18:52:34.072083 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:37.073050 (+3000967us) service_pool.cc:227] Handling call
0810 18:52:37.084481 (+ 11431us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:38.065438 (+980957us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:38.194461 (+129023us) spinlock_profiling.cc:243] Waited 934 ms on lock 0x7b540020b054. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:38.194742 (+ 281us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2148629760}
W20260810 18:52:38.194331 17775 rpcz_store.cc:269] 0810 18:52:36.749252 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:36.784059 (+ 34807us) service_pool.cc:227] Handling call
0810 18:52:38.193403 (+1409344us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54006a27d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:38.193667 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2406210816}
I20260810 18:52:38.169953 16573 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:52:38.170316 21038 raft_consensus.cc:729] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:38.067111 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260810 18:52:38.222800 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 141447us
I20260810 18:52:38.155182 16636 rpcz_store.cc:276] 0810 18:52:36.005938 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:36.655305 (+649367us) service_pool.cc:227] Handling call
0810 18:52:36.655691 (+ 386us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:36.667250 (+ 11559us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:36.667289 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:36.675952 (+ 8663us) log.cc:842] Serialized 26 byte log entry
0810 18:52:37.487412 (+811460us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:37.487451 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:37.487524 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:37.986440 (+498916us) raft_consensus.cc:1671] finished
0810 18:52:38.009229 (+ 22789us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:38.057121 (+ 47892us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11002,"raft.queue_time_us":15950,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":14171,"thread_start_us":741376,"threads_started":1,"wal-append.queue_time_us":1019869}
W20260810 18:52:38.252120 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 91871.4us
I20260810 18:52:38.255812 16496 raft_consensus.cc:1261] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260810 18:52:38.256067 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:38.257056 16626 raft_consensus.cc:1226] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 7. Current term is 8. Ops: []
I20260810 18:52:38.187018 21296 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:38.260701 17786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55586 (request call id 448) took 1035 ms. Trace:
I20260810 18:52:38.275652 17786 rpcz_store.cc:276] 0810 18:52:37.212171 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:37.212364 (+ 193us) service_pool.cc:227] Handling call
0810 18:52:38.234144 (+1021780us) spinlock_profiling.cc:243] Waited 886 ms on lock 0x7b54004f06d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:38.247339 (+ 13195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2036794752}
I20260810 18:52:38.267468 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:52:38.314167 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 58230.1us
I20260810 18:52:38.323385 16501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52352 (request call id 1657) took 1587 ms. Trace:
I20260810 18:52:38.323926 16501 rpcz_store.cc:276] 0810 18:52:36.661441 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:37.165992 (+504551us) service_pool.cc:227] Handling call
0810 18:52:38.234376 (+1068384us) spinlock_profiling.cc:243] Waited 858 ms on lock 0x7b54001cc0d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:38.248715 (+ 14339us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1974476672}
W20260810 18:52:38.324877 16701 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.024s sys 0.039s
W20260810 18:52:38.325349 16701 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.638s user 0.024s sys 0.040s
W20260810 18:52:38.325855 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 639307us
I20260810 18:52:38.328056 16626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 2071) took 1269 ms. Trace:
I20260810 18:52:38.328497 16626 rpcz_store.cc:276] 0810 18:52:37.058773 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:37.314220 (+255447us) service_pool.cc:227] Handling call
0810 18:52:37.332429 (+ 18209us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:38.257611 (+925182us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:38.327658 (+ 70047us) spinlock_profiling.cc:243] Waited 897 ms on lock 0x7b54002a4754. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:38.327932 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2062713728}
I20260810 18:52:38.333411 16502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:52330 (request call id 858) took 1347 ms. Trace:
I20260810 18:52:38.333899 16502 rpcz_store.cc:276] 0810 18:52:36.985570 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:38.136851 (+1151281us) service_pool.cc:227] Handling call
0810 18:52:38.333258 (+196407us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:38.335151 16505 raft_consensus.cc:2454] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 7.
W20260810 18:52:38.337797 21255 consensus_meta.cc:287] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.685s user 0.016s sys 0.002s
I20260810 18:52:38.338589 16749 raft_consensus.cc:1226] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:38.340242 21200 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 9 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:38.340453 16749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43726 (request call id 1572) took 2317 ms. Trace:
I20260810 18:52:38.340874 16749 rpcz_store.cc:276] 0810 18:52:36.022691 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:37.188266 (+1165575us) service_pool.cc:227] Handling call
0810 18:52:38.267381 (+1079115us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:38.339115 (+ 71734us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:38.340162 (+ 1047us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540029ca50. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:38.340384 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2434733952}
I20260810 18:52:38.341601 16750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43726 (request call id 1580) took 2230 ms. Trace:
I20260810 18:52:38.341990 16750 rpcz_store.cc:276] 0810 18:52:36.111216 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:37.198908 (+1087692us) service_pool.cc:227] Handling call
0810 18:52:37.228325 (+ 29417us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:38.120496 (+892171us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:38.341266 (+220770us) spinlock_profiling.cc:243] Waited 857 ms on lock 0x7b54002ed1d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:38.341539 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1970248448}
W20260810 18:52:38.343349 16745 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43738 (request call id 2095) took 2092 ms (client timeout 2138 ms). Trace:
W20260810 18:52:38.343832 16745 rpcz_store.cc:269] 0810 18:52:36.182674 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:36.683157 (+500483us) service_pool.cc:227] Handling call
0810 18:52:38.242556 (+1559399us) spinlock_profiling.cc:243] Waited 836 ms on lock 0x7b540029ca54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:38.275300 (+ 32744us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1922062720}
I20260810 18:52:38.347903 16496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1591) took 1587 ms. Trace:
I20260810 18:52:38.348311 16496 rpcz_store.cc:276] 0810 18:52:36.760823 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:38.130863 (+1370040us) service_pool.cc:227] Handling call
0810 18:52:38.144398 (+ 13535us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:38.256392 (+111994us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:38.347635 (+ 91243us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b54000f7154. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:38.347811 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":236615552}
I20260810 18:52:38.349473 16573 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:38.349905 21469 consensus_queue.cc:1035] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:52:38.352798 21296 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:38.353241 21461 raft_consensus.cc:2726] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260810 18:52:38.353621 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 78370us
W20260810 18:52:38.356772 21359 consensus_meta.cc:287] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 1.576s user 0.000s sys 0.031s
I20260810 18:52:38.357357 21359 raft_consensus.cc:2726] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:38.358697 21170 raft_consensus.cc:2781] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:38.375198 21170 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:38.375715 21170 raft_consensus.cc:3037] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:38.317444 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:38.359709 21380 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 9 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:38.360416 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 146552us
W20260810 18:52:38.387980 21450 log.cc:924] Time spent T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260810 18:52:38.392237 16501 raft_consensus.cc:1261] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260810 18:52:38.393683 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:38.396544 21154 consensus_meta.cc:287] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 1.299s user 0.011s sys 0.015s
I20260810 18:52:38.397240 21154 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:38.361469 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:38.258232 21038 consensus_queue.cc:260] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:38.364930 21271 consensus_meta.cc:287] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.787s user 0.006s sys 0.009s
W20260810 18:52:38.365080 20994 consensus_meta.cc:287] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 1.516s user 0.016s sys 0.012s
I20260810 18:52:38.410370 16342 catalog_manager.cc:5495] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 3 to 6, leader changed from 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 6 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
W20260810 18:52:38.365502 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 59773.4us
I20260810 18:52:38.411449 21271 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:38.412325 16448 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:38.413098 21391 raft_consensus.cc:482] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:38.412436 16342 catalog_manager.cc:5495] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 1 to 6, leader changed from d304ac47fdf846de81337057a4ef7bea (127.14.32.193) to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 6 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:52:38.413643 21391 raft_consensus.cc:504] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:38.414495 21038 raft_consensus.cc:3037] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:38.414753 16342 catalog_manager.cc:5495] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "831925dd8b644ebab7eb0d672a94eddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:52:38.417826 21391 leader_election.cc:288] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:38.366622 21226 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:38.399703 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:38.419586 16496 raft_consensus.cc:3037] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:38.422353 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
W20260810 18:52:38.360854 21202 consensus_meta.cc:287] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.631s user 0.013s sys 0.005s
I20260810 18:52:38.423637 21202 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:38.426357 21202 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:38.405411 16505 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52352 (request call id 1659) took 1738 ms. Trace:
W20260810 18:52:38.405259 17770 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55602 (request call id 414) took 1740 ms (client timeout 1554 ms). Trace:
I20260810 18:52:38.427130 16505 rpcz_store.cc:276] 0810 18:52:36.666604 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:37.184789 (+518185us) service_pool.cc:227] Handling call
0810 18:52:38.405205 (+1220416us) spinlock_profiling.cc:243] Waited 737 ms on lock 0x7b5400332f54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:38.405354 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1696233344}
I20260810 18:52:38.405934 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:38.427219 17770 rpcz_store.cc:269] 0810 18:52:36.664291 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:36.664464 (+ 173us) service_pool.cc:227] Handling call
0810 18:52:38.404817 (+1740353us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54006b1554. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:38.405024 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2911447296}
I20260810 18:52:38.430784 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:38.431881 16745 raft_consensus.cc:2379] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 8: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
W20260810 18:52:38.433177 16506 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52338 (request call id 1567) took 2106 ms (client timeout 2309 ms). Trace:
I20260810 18:52:38.433302 21191 consensus_queue.cc:1046] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260810 18:52:38.433648 16506 rpcz_store.cc:269] 0810 18:52:36.327045 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:36.328354 (+ 1309us) service_pool.cc:227] Handling call
0810 18:52:38.432913 (+2104559us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b5400022454. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:38.433101 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3404070656}
I20260810 18:52:38.434599 21191 raft_consensus.cc:3032] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 4 LEADER]: Stepping down as leader of term 4
I20260810 18:52:38.434922 21191 raft_consensus.cc:729] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
W20260810 18:52:38.408607 21062 consensus_meta.cc:287] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 1.409s user 0.015s sys 0.013s
I20260810 18:52:38.435938 21062 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:38.436106 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:38.437115 17781 raft_consensus.cc:2454] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:38.437997 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
W20260810 18:52:38.438540 17781 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55586 (request call id 446) took 1357 ms (client timeout 1535 ms). Trace:
W20260810 18:52:38.438978 17781 rpcz_store.cc:269] 0810 18:52:37.081503 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:37.081647 (+ 144us) service_pool.cc:227] Handling call
0810 18:52:38.438307 (+1356660us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b540072f6d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:38.438434 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2953855872}
I20260810 18:52:38.439934 21461 consensus_queue.cc:1035] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260810 18:52:38.405296 16450 leader_election.cc:302] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:52:38.441010 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 86204.3us
I20260810 18:52:38.442140 16573 leader_election.cc:398] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Vote denied by peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791) with higher term. Message: Invalid argument: T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 4. Current term is 5.
I20260810 18:52:38.443250 16573 leader_election.cc:401] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260810 18:52:38.443838 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:38.444679 16750 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:38.446374 16573 leader_election.cc:398] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Vote denied by peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791) with higher term. Message: Invalid argument: T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 4. Current term is 5.
I20260810 18:52:38.446727 16573 leader_election.cc:401] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260810 18:52:38.447335 21226 raft_consensus.cc:2726] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791) with higher term. Message: Invalid argument: T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 4. Current term is 5.
I20260810 18:52:38.448588 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:38.449441 16342 catalog_manager.cc:5495] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 5 to 6, leader changed from 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194) to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 6 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:52:38.451658 16342 catalog_manager.cc:5495] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "831925dd8b644ebab7eb0d672a94eddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:52:38.452739 21296 consensus_queue.cc:1046] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260810 18:52:38.453795 21296 raft_consensus.cc:3032] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 8 LEADER]: Stepping down as leader of term 8
I20260810 18:52:38.454245 21296 raft_consensus.cc:729] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:38.438127 21062 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:38.450039 16745 raft_consensus.cc:3037] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Advancing to term 8
W20260810 18:52:38.405190 21290 consensus_meta.cc:287] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 1.040s user 0.015s sys 0.005s
W20260810 18:52:38.410926 16701 leader_election.cc:334] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.574s (SENT)
I20260810 18:52:38.436056 21191 consensus_queue.cc:260] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:38.469583 16742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43726 (request call id 1562) took 2450 ms. Trace:
I20260810 18:52:38.479388 21004 raft_consensus.cc:2781] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election won for term 7
I20260810 18:52:38.479664 21272 raft_consensus.cc:2781] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:38.480922 16740 raft_consensus.cc:2379] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 8: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:38.482188 21226 raft_consensus.cc:2726] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791) with higher term. Message: Invalid argument: T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 4. Current term is 5.
I20260810 18:52:38.483760 21088 raft_consensus.cc:2726] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260810 18:52:38.483923 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 79051.4us
I20260810 18:52:38.484611 21088 consensus_queue.cc:1046] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260810 18:52:38.485955 21088 raft_consensus.cc:3032] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Stepping down as leader of term 7
I20260810 18:52:38.486369 21088 raft_consensus.cc:729] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:38.490988 16748 raft_consensus.cc:1226] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 4. Ops: []
I20260810 18:52:38.491530 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:38.491590 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:38.492787 16636 raft_consensus.cc:2396] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260810 18:52:38.492542 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:38.494333 16633 raft_consensus.cc:1226] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 7. Current term is 8. Ops: []
I20260810 18:52:38.494860 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:38.495215 21191 raft_consensus.cc:3037] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:38.496805 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:38.488005 16701 leader_election.cc:302] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:38.499630 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:38.499768 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:38.500072 20994 raft_consensus.cc:2726] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:38.500456 16626 raft_consensus.cc:3037] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:38.501391 16636 pending_rounds.cc:76] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb: Aborting all ops after (but not including) 5
I20260810 18:52:38.477967 16742 rpcz_store.cc:276] 0810 18:52:36.018963 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:37.178479 (+1159516us) service_pool.cc:227] Handling call
0810 18:52:37.180332 (+ 1853us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:37.180584 (+ 252us) raft_consensus.cc:1520] Early marking committed up to index 5
0810 18:52:38.159713 (+979129us) log.cc:842] Serialized 14 byte log entry
0810 18:52:38.304360 (+144647us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:38.314762 (+ 10402us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:38.317748 (+ 2986us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:38.460840 (+143092us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:38.469502 (+ 8662us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3744,"raft.queue_time_us":20622,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":47542,"thread_start_us":972260,"threads_started":2}
I20260810 18:52:38.501703 16636 pending_rounds.cc:97] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260810 18:52:38.502166 16636 raft_consensus.cc:2866] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:52:38.503048 21154 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:38.501410 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 90552.8us
I20260810 18:52:38.500406 16633 raft_consensus.cc:2454] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
I20260810 18:52:38.460639 21271 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:38.514868 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
W20260810 18:52:38.520025 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 73864.3us
I20260810 18:52:38.521481 16622 raft_consensus.cc:2396] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260810 18:52:38.521629 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:38.522008 17786 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:38.522526 16633 raft_consensus.cc:3037] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:38.511955 21004 raft_consensus.cc:686] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:38.526686 21088 consensus_queue.cc:260] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:38.526039 16575 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.594s user 0.028s sys 0.036s
W20260810 18:52:38.536391 16575 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.877s user 0.028s sys 0.037s
I20260810 18:52:38.537390 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:38.538436 16624 raft_consensus.cc:2379] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 8: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:38.539716 16500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1646) took 1889 ms. Trace:
I20260810 18:52:38.540485 16632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 2063) took 2419 ms. Trace:
I20260810 18:52:38.540053 21004 consensus_queue.cc:237] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:38.540129 16500 rpcz_store.cc:276] 0810 18:52:36.650222 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:37.332823 (+682601us) service_pool.cc:227] Handling call
0810 18:52:37.375923 (+ 43100us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:37.376171 (+ 248us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:37.376197 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:37.376628 (+ 431us) log.cc:842] Serialized 26 byte log entry
0810 18:52:38.251351 (+874723us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:38.304812 (+ 53461us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:38.315249 (+ 10437us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:38.539433 (+224184us) raft_consensus.cc:1671] finished
0810 18:52:38.539456 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:38.539634 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":771107,"threads_started":1,"wal-append.queue_time_us":1160384}
I20260810 18:52:38.540982 16632 rpcz_store.cc:276] 0810 18:52:36.121492 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:36.790881 (+669389us) service_pool.cc:227] Handling call
0810 18:52:36.807555 (+ 16674us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:36.808212 (+ 657us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:36.808331 (+ 119us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:36.833504 (+ 25173us) log.cc:842] Serialized 26 byte log entry
0810 18:52:37.656078 (+822574us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:37.656135 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:37.656231 (+ 96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:38.540046 (+883815us) raft_consensus.cc:1671] finished
0810 18:52:38.540093 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:38.540313 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":105,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":7526,"thread_start_us":781182,"threads_started":1,"wal-append.queue_time_us":1684944}
I20260810 18:52:38.545030 17787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55586 (request call id 447) took 1451 ms. Trace:
I20260810 18:52:38.546089 16447 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:38.547524 21057 raft_consensus.cc:2781] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader election won for term 8
I20260810 18:52:38.548218 21057 raft_consensus.cc:686] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 8 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:38.549662 21057 consensus_queue.cc:237] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:38.551651 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 894001us
I20260810 18:52:38.545473 17787 rpcz_store.cc:276] 0810 18:52:37.093288 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:37.093430 (+ 142us) service_pool.cc:227] Handling call
0810 18:52:37.101394 (+ 7964us) raft_consensus.cc:1457] Updating replica for 2 ops
0810 18:52:37.101890 (+ 496us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:37.101928 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0810 18:52:37.102585 (+ 657us) log.cc:842] Serialized 52 byte log entry
0810 18:52:37.685151 (+582566us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:37.685205 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:37.685317 (+ 112us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:38.544512 (+859195us) raft_consensus.cc:1671] finished
0810 18:52:38.544558 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:38.544824 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1425,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":120,"thread_start_us":582078,"threads_started":1,"wal-append.queue_time_us":1419621}
I20260810 18:52:38.552663 16575 leader_election.cc:302] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:38.553920 21460 raft_consensus.cc:2781] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:38.554432 21460 raft_consensus.cc:482] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:38.554917 21460 raft_consensus.cc:3037] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:38.562759 16504 raft_consensus.cc:2454] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
I20260810 18:52:38.569402 16626 raft_consensus.cc:2396] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260810 18:52:38.576265 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:38.579916 16573 leader_election.cc:302] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:38.582208 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:38.587445 16496 raft_consensus.cc:2454] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 9.
I20260810 18:52:38.588845 16742 raft_consensus.cc:2379] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 5: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:38.530153 21296 consensus_queue.cc:260] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:38.591305 21290 raft_consensus.cc:2741] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
W20260810 18:52:38.601563 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 93300.2us
W20260810 18:52:38.602080 16622 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 2051) took 2574 ms (client timeout 2170 ms). Trace:
I20260810 18:52:38.602494 16632 raft_consensus.cc:1261] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260810 18:52:38.608338 16575 leader_election.cc:302] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
W20260810 18:52:38.613747 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 67822.3us
W20260810 18:52:38.615787 21459 log.cc:924] Time spent T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.101s user 0.000s sys 0.005s
W20260810 18:52:38.611923 16622 rpcz_store.cc:269] 0810 18:52:36.025524 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:36.536790 (+511266us) service_pool.cc:227] Handling call
0810 18:52:38.589032 (+2052242us) spinlock_profiling.cc:243] Waited 1.88 s on lock 0x7b54003683d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:38.600292 (+ 11260us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4318434048}
I20260810 18:52:38.616850 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
W20260810 18:52:38.622336 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 52794.8us
I20260810 18:52:38.625180 16749 raft_consensus.cc:2379] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 7: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:38.625216 16748 raft_consensus.cc:2454] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
W20260810 18:52:38.626824 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 47434.6us
I20260810 18:52:38.624802 21226 consensus_queue.cc:1035] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260810 18:52:38.633996 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:38.634792 16622 raft_consensus.cc:3037] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:38.642105 16750 raft_consensus.cc:2454] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 9.
I20260810 18:52:38.644057 17727 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:38.645179 21272 raft_consensus.cc:482] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:38.645551 16743 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af"
dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
from {username='slave'} at 127.0.0.1:43738
I20260810 18:52:38.646252 16743 raft_consensus.cc:482] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:52:38.646574 16743 raft_consensus.cc:3037] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:38.645768 21272 raft_consensus.cc:3037] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:38.648934 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:38.652269 17727 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:38.650471 16575 leader_election.cc:302] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:38.652037 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:38.645118 21437 raft_consensus.cc:2781] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:38.653971 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:38.657142 16747 raft_consensus.cc:1226] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 7. Current term is 8. Ops: []
I20260810 18:52:38.656790 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:38.659622 17787 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:38.659786 16623 raft_consensus.cc:1261] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260810 18:52:38.657737 17781 raft_consensus.cc:1261] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260810 18:52:38.663805 21437 raft_consensus.cc:686] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 6 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
W20260810 18:52:38.657904 16740 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37716 (request call id 418) took 2718 ms (client timeout 2630 ms). Trace:
W20260810 18:52:38.666543 16740 rpcz_store.cc:269] 0810 18:52:35.937150 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:36.843727 (+906577us) service_pool.cc:227] Handling call
0810 18:52:38.650441 (+1806714us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b5400140d54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:38.655804 (+ 5363us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3754630016}
I20260810 18:52:38.669523 21308 raft_consensus.cc:2781] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Leader election won for term 9
I20260810 18:52:38.670174 21308 raft_consensus.cc:686] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 9 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:38.669730 21437 consensus_queue.cc:237] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:38.671240 21226 raft_consensus.cc:2781] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:38.670871 17727 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:52:38.673583 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 48429.8us
I20260810 18:52:38.661659 21296 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Advancing to term 9
W20260810 18:52:38.677728 16695 leader_election.cc:334] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387): Timed out: RequestConsensusVote RPC to 127.14.32.196:33387 timed out after 1.614s (SENT)
I20260810 18:52:38.671685 21226 raft_consensus.cc:482] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:38.682036 16631 raft_consensus.cc:1261] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260810 18:52:38.682381 21226 raft_consensus.cc:3037] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:38.679725 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
W20260810 18:52:38.686216 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 381551us
I20260810 18:52:38.690043 16621 raft_consensus.cc:1261] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260810 18:52:38.690445 16742 raft_consensus.cc:2379] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 8: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
W20260810 18:52:38.689224 16695 leader_election.cc:334] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791): Timed out: RequestConsensusVote RPC to 127.14.32.193:40791 timed out after 1.614s (SENT)
I20260810 18:52:38.691756 16501 raft_consensus.cc:2396] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260810 18:52:38.690646 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:38.694294 16740 raft_consensus.cc:3037] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:38.693748 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:38.687238 21290 raft_consensus.cc:2781] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260810 18:52:38.691260 16695 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:38.697388 16631 raft_consensus.cc:1261] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260810 18:52:38.700680 17787 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 8.
I20260810 18:52:38.701697 21434 raft_consensus.cc:2726] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260810 18:52:38.693693 21477 log.cc:924] Time spent T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
W20260810 18:52:38.715636 21475 log.cc:924] Time spent T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.061s user 0.000s sys 0.001s
I20260810 18:52:38.717042 16495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1650) took 2062 ms. Trace:
I20260810 18:52:38.717424 16495 rpcz_store.cc:276] 0810 18:52:36.654285 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:37.889173 (+1234888us) service_pool.cc:227] Handling call
0810 18:52:37.890164 (+ 991us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:37.890802 (+ 638us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:37.890864 (+ 62us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:38.143219 (+252355us) log.cc:842] Serialized 26 byte log entry
0810 18:52:38.222317 (+ 79098us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:38.222365 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:38.222446 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:38.716778 (+494332us) raft_consensus.cc:1671] finished
0810 18:52:38.716800 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:38.716958 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":56355,"threads_started":1,"wal-append.queue_time_us":464468}
I20260810 18:52:38.709187 21088 raft_consensus.cc:3037] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:38.715629 21290 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:38.724983 16504 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52338 (request call id 1576) took 2283 ms (client timeout 1634 ms). Trace:
I20260810 18:52:38.727509 16508 raft_consensus.cc:1226] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260810 18:52:38.727577 16622 raft_consensus.cc:2454] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
W20260810 18:52:38.725498 16504 rpcz_store.cc:269] 0810 18:52:36.441640 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:36.606272 (+164632us) service_pool.cc:227] Handling call
0810 18:52:38.724727 (+2118455us) spinlock_profiling.cc:243] Waited 1.75 s on lock 0x7b54000e0bd4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:38.724902 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4025767168}
I20260810 18:52:38.729369 16750 raft_consensus.cc:3037] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:38.731695 16745 raft_consensus.cc:2454] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 8.
I20260810 18:52:38.732283 16743 raft_consensus.cc:504] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:38.732749 21154 raft_consensus.cc:2726] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260810 18:52:38.740614 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:38.743739 16745 raft_consensus.cc:3037] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:38.745697 21272 raft_consensus.cc:504] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:38.747622 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 54234.8us
W20260810 18:52:38.752138 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 62950.8us
I20260810 18:52:38.736990 16624 raft_consensus.cc:2454] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 8.
I20260810 18:52:38.753010 16508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1660) took 2086 ms. Trace:
I20260810 18:52:38.671250 21308 consensus_queue.cc:237] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:38.737432 17727 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:38.726472 21290 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Advancing to term 10
I20260810 18:52:38.742743 16747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43726 (request call id 1574) took 2716 ms. Trace:
I20260810 18:52:38.736599 16740 raft_consensus.cc:2454] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
I20260810 18:52:38.753711 16749 raft_consensus.cc:1261] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260810 18:52:38.753453 16508 rpcz_store.cc:276] 0810 18:52:36.666885 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:38.139220 (+1472335us) service_pool.cc:227] Handling call
0810 18:52:38.726043 (+586823us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:38.728568 (+ 2525us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:38.751288 (+ 22720us) spinlock_profiling.cc:243] Waited 586 ms on lock 0x7b54000e0bd0. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:38.752940 (+ 1652us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1348356480}
W20260810 18:52:38.760648 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 73313.3us
I20260810 18:52:38.759769 16633 raft_consensus.cc:2396] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260810 18:52:38.754290 17786 raft_consensus.cc:2454] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 9.
I20260810 18:52:38.761598 16573 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:38.758837 16745 raft_consensus.cc:2454] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
I20260810 18:52:38.760142 16747 rpcz_store.cc:276] 0810 18:52:36.025967 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:37.198618 (+1172651us) service_pool.cc:227] Handling call
0810 18:52:38.655573 (+1456955us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:38.657606 (+ 2033us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:38.742524 (+ 84918us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b5400140d50. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:38.742685 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3346744576}
I20260810 18:52:38.740859 21460 raft_consensus.cc:504] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:38.769516 21460 leader_election.cc:288] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:38.771315 16626 raft_consensus.cc:1226] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:38.784392 16575 leader_election.cc:302] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:38.784924 16573 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:38.785815 21461 raft_consensus.cc:2726] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:38.785950 16575 leader_election.cc:398] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Vote denied by peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171) with higher term. Message: Invalid argument: T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6.
I20260810 18:52:38.786279 16575 leader_election.cc:401] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260810 18:52:38.786454 21461 raft_consensus.cc:2781] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:38.787253 21380 raft_consensus.cc:2781] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260810 18:52:38.787642 21380 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:38.787956 21380 raft_consensus.cc:3037] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:38.788810 21469 raft_consensus.cc:2726] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171) with higher term. Message: Invalid argument: T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6.
I20260810 18:52:38.786890 21461 raft_consensus.cc:482] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:38.789659 21461 raft_consensus.cc:3037] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:38.790596 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:38.791402 16573 leader_election.cc:302] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:38.789556 16447 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:38.791925 16748 raft_consensus.cc:2379] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 8: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:38.777182 16742 raft_consensus.cc:1261] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260810 18:52:38.799618 21469 raft_consensus.cc:2781] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:38.802670 21469 raft_consensus.cc:482] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:38.803133 21469 raft_consensus.cc:3037] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Advancing to term 7
W20260810 18:52:38.811467 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 44690.4us
I20260810 18:52:38.812527 16448 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:38.814430 21359 raft_consensus.cc:2781] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260810 18:52:38.815222 21359 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:38.815740 21359 raft_consensus.cc:3037] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:38.817255 16575 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:38.818697 16575 leader_election.cc:302] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:38.821195 21460 raft_consensus.cc:2781] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:38.821762 21460 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:38.822197 21460 raft_consensus.cc:3037] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:38.823939 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 9 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:38.824898 16501 raft_consensus.cc:2396] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260810 18:52:38.824465 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:38.827709 16508 raft_consensus.cc:1226] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:38.835578 16743 leader_election.cc:288] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:38.766315 21191 raft_consensus.cc:2781] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader election won for term 8
I20260810 18:52:38.853150 16740 raft_consensus.cc:1226] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:38.854630 16695 leader_election.cc:398] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 election: Vote denied by peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387) with higher term. Message: Invalid argument: T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 8. Current term is 9.
I20260810 18:52:38.856740 16695 leader_election.cc:401] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
W20260810 18:52:38.857815 16701 leader_election.cc:334] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.101s (SENT)
I20260810 18:52:38.863387 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:38.863909 16750 raft_consensus.cc:2454] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
I20260810 18:52:38.865656 16749 raft_consensus.cc:1226] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:38.833913 21088 raft_consensus.cc:2726] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:38.863490 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:38.867527 16573 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:38.869395 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:38.863569 16701 leader_election.cc:302] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:38.871891 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 73500.4us
I20260810 18:52:38.872674 16504 raft_consensus.cc:2379] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 7: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
W20260810 18:52:38.873513 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 54896.5us
I20260810 18:52:38.874302 16508 raft_consensus.cc:2379] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 6: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:38.875720 16501 raft_consensus.cc:2379] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 7: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:38.886270 21272 leader_election.cc:288] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:38.886746 21359 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:38.889489 21359 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:38.880358 21057 consensus_queue.cc:1035] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20260810 18:52:38.876159 21444 consensus_queue.cc:1035] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20260810 18:52:38.893790 21462 consensus_queue.cc:1035] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 18:52:38.894624 21391 raft_consensus.cc:2726] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:38.892417 21062 consensus_queue.cc:1046] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260810 18:52:38.932120 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 79055.1us
I20260810 18:52:38.932950 16508 raft_consensus.cc:1226] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:38.936898 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:38.943306 16695 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:38.880350 21154 raft_consensus.cc:2726] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387) with higher term. Message: Invalid argument: T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 8. Current term is 9.
I20260810 18:52:38.877043 21191 raft_consensus.cc:686] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 8 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:38.947108 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:38.950783 16495 raft_consensus.cc:1226] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260810 18:52:38.951203 16504 raft_consensus.cc:2454] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
I20260810 18:52:38.932926 21088 consensus_queue.cc:1035] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
W20260810 18:52:38.956367 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 113270us
I20260810 18:52:38.957873 21490 raft_consensus.cc:2781] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:38.958416 21490 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:38.958809 21490 raft_consensus.cc:3037] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:38.960510 16448 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:38.960179 21191 consensus_queue.cc:237] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:38.912595 21437 consensus_queue.cc:1046] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
I20260810 18:52:38.963876 21437 raft_consensus.cc:3032] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:38.964329 21437 raft_consensus.cc:729] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:38.964620 21290 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:38.965278 21437 consensus_queue.cc:260] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:38.966775 21437 raft_consensus.cc:3037] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Advancing to term 6
W20260810 18:52:38.968103 21484 log.cc:924] Time spent T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.121s user 0.000s sys 0.001s
I20260810 18:52:38.971184 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:38.999897 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:38.973850 21469 raft_consensus.cc:504] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:38.978329 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:38.979377 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:38.980432 16505 raft_consensus.cc:1226] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:38.981271 16502 raft_consensus.cc:1226] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:38.981915 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:38.983436 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:38.988938 17786 raft_consensus.cc:1261] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260810 18:52:38.989197 21380 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:38.989609 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:39.006611 16506 raft_consensus.cc:3037] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Advancing to term 8
W20260810 18:52:38.993284 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 72989.2us
I20260810 18:52:38.993772 21154 raft_consensus.cc:2741] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260810 18:52:38.996340 16448 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:38.960228 16508 raft_consensus.cc:1226] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260810 18:52:39.006361 16495 raft_consensus.cc:2379] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 8: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:38.991930 21308 raft_consensus.cc:3032] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 6 LEADER]: Stepping down as leader of term 6
I20260810 18:52:39.007958 16501 raft_consensus.cc:2379] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 5: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:39.010929 21462 raft_consensus.cc:2781] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election won for term 7
W20260810 18:52:39.019037 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 67052.9us
I20260810 18:52:39.021592 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:39.024982 16505 raft_consensus.cc:1226] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:39.022579 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:39.029032 16495 raft_consensus.cc:2379] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 7: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:39.024230 21462 raft_consensus.cc:686] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:39.032480 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:39.033811 17771 raft_consensus.cc:2379] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 8: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:39.035632 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:39.033689 16501 raft_consensus.cc:1261] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260810 18:52:39.036293 16508 raft_consensus.cc:2454] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:39.037411 16507 raft_consensus.cc:2354] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for earlier term 6. Current term is 7.
I20260810 18:52:39.033226 21462 consensus_queue.cc:237] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:39.032612 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:39.011713 21308 raft_consensus.cc:729] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:39.012296 21290 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 10 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:39.042342 16500 raft_consensus.cc:1226] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 8. Current term is 9. Ops: []
I20260810 18:52:38.994777 21444 consensus_queue.cc:1035] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20260810 18:52:39.034258 21057 raft_consensus.cc:2781] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader election won for term 8
I20260810 18:52:39.055871 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:39.061165 16633 raft_consensus.cc:2396] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260810 18:52:39.065459 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:39.064369 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:39.069895 16500 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:39.073374 17771 raft_consensus.cc:2354] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for earlier term 6. Current term is 7.
I20260810 18:52:39.072326 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:39.074991 16632 raft_consensus.cc:3037] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Advancing to term 9
I20260810 18:52:39.041304 21038 consensus_queue.cc:1046] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260810 18:52:39.075565 16695 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:39.079237 16748 raft_consensus.cc:1261] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260810 18:52:39.085404 16750 raft_consensus.cc:1261] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260810 18:52:39.088406 21272 raft_consensus.cc:2781] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Leader election won for term 9
I20260810 18:52:39.088660 21495 consensus_queue.cc:1046] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:52:39.089100 21272 raft_consensus.cc:686] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:39.096557 16745 raft_consensus.cc:3032] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Stepping down as leader of term 6
I20260810 18:52:39.096963 16745 raft_consensus.cc:729] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:39.074379 21057 raft_consensus.cc:686] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 8 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:39.104475 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:52:39.114478 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 45028.3us
W20260810 18:52:39.107954 16345 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37022 (request call id 214) took 16677 ms (client timeout 15000 ms). Trace:
I20260810 18:52:39.115447 16496 raft_consensus.cc:2454] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 9.
I20260810 18:52:39.123639 16740 raft_consensus.cc:1226] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 8. Current term is 9. Ops: []
I20260810 18:52:39.068639 21391 consensus_queue.cc:1035] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.039s
I20260810 18:52:39.071949 21290 consensus_queue.cc:1046] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20260810 18:52:39.136615 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:39.135505 21088 raft_consensus.cc:482] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:39.142391 21272 consensus_queue.cc:237] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:39.149255 21271 raft_consensus.cc:482] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:39.152222 21271 raft_consensus.cc:504] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:39.151536 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:39.154259 21271 leader_election.cc:288] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:39.155120 21491 log.cc:924] Time spent T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20260810 18:52:39.151602 21088 raft_consensus.cc:504] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:52:39.157469 21088 leader_election.cc:288] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:39.151476 21493 raft_consensus.cc:2726] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260810 18:52:39.165731 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 137534us
W20260810 18:52:39.167431 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 86275us
W20260810 18:52:39.167891 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 94630.5us
W20260810 18:52:39.147791 21170 consensus_meta.cc:287] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.771s user 0.011s sys 0.003s
I20260810 18:52:39.164039 16745 consensus_queue.cc:260] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:39.104118 21226 raft_consensus.cc:504] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:39.105017 21461 raft_consensus.cc:504] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:39.098152 21308 consensus_queue.cc:260] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:39.186280 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:39.168036 21380 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 9 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:39.184666 17724 leader_election.cc:302] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:39.183532 16695 leader_election.cc:302] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:39.199572 16745 raft_consensus.cc:3037] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:39.199745 16626 raft_consensus.cc:1226] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:39.201263 16500 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 8.
I20260810 18:52:39.205057 16633 raft_consensus.cc:1261] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260810 18:52:39.210755 16624 raft_consensus.cc:1261] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260810 18:52:39.174517 21469 leader_election.cc:288] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:39.211329 16624 pending_rounds.cc:76] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb: Aborting all ops after (but not including) 3
I20260810 18:52:39.219030 16624 pending_rounds.cc:97] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260810 18:52:39.219370 16624 raft_consensus.cc:2866] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:52:39.159638 21057 consensus_queue.cc:237] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:39.222879 16745 raft_consensus.cc:1261] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260810 18:52:39.184962 21170 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:39.215942 21191 raft_consensus.cc:2726] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:39.229596 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:39.223290 16745 pending_rounds.cc:76] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be: Aborting all ops after (but not including) 3
I20260810 18:52:39.231760 16745 pending_rounds.cc:97] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260810 18:52:39.232044 16745 raft_consensus.cc:2866] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:52:39.221192 21462 consensus_queue.cc:1035] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.013s
I20260810 18:52:39.235402 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:39.241863 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:39.227269 21391 raft_consensus.cc:2781] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:39.225907 21308 raft_consensus.cc:3037] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:39.246325 16621 raft_consensus.cc:1226] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 10 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 9. Current term is 10. Ops: []
W20260810 18:52:39.239951 17726 leader_election.cc:334] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 3.225s (SENT)
I20260810 18:52:39.246841 16626 raft_consensus.cc:1261] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260810 18:52:39.239415 16506 raft_consensus.cc:2454] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 8.
I20260810 18:52:39.251500 16632 raft_consensus.cc:2454] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 9.
W20260810 18:52:39.122121 16345 rpcz_store.cc:269] 0810 18:52:22.285717 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:22.286624 (+ 907us) service_pool.cc:227] Handling call
0810 18:52:38.948915 (+16662291us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 18:52:29.402418 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 18:52:29.403058 (+ 640us) write_op.cc:432] Acquiring schema lock in shared mode
0810 18:52:29.403130 (+ 72us) write_op.cc:435] Acquired schema lock
0810 18:52:29.403147 (+ 17us) tablet.cc:660] Decoding operations
0810 18:52:29.536219 (+133072us) write_op.cc:620] Acquiring the partition lock for write op
0810 18:52:29.536337 (+ 118us) write_op.cc:641] Partition lock acquired for write op
0810 18:52:29.536368 (+ 31us) tablet.cc:683] Acquiring locks for 30 operations
0810 18:52:29.538754 (+ 2386us) tablet.cc:699] Row locks acquired
0810 18:52:29.538789 (+ 35us) write_op.cc:260] PREPARE: finished
0810 18:52:29.538965 (+ 176us) write_op.cc:270] Start()
0810 18:52:29.588454 (+ 49489us) write_op.cc:276] Timestamp: P: 1786387949538900 usec, L: 0
0810 18:52:29.601697 (+ 13243us) op_driver.cc:331] REPLICATION: starting
0810 18:52:29.631759 (+ 30062us) log.cc:842] Serialized 11746 byte log entry
0810 18:52:30.129869 (+498110us) op_driver.cc:444] REPLICATION: finished
0810 18:52:30.355466 (+225597us) write_op.cc:301] APPLY: starting
0810 18:52:30.355576 (+ 110us) tablet.cc:1365] starting BulkCheckPresence
0810 18:52:30.356142 (+ 566us) tablet.cc:1368] finished BulkCheckPresence
0810 18:52:30.356281 (+ 139us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 18:52:30.475378 (+119097us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 18:52:30.789016 (+313638us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=30
0810 18:52:30.795417 (+ 6401us) write_op.cc:312] APPLY: finished
0810 18:52:31.041101 (+245684us) log.cc:842] Serialized 200 byte log entry
0810 18:52:31.073370 (+ 32269us) write_op.cc:489] Releasing partition, row and schema locks
0810 18:52:31.096279 (+ 22909us) write_op.cc:454] Released schema lock
0810 18:52:31.246967 (+150688us) write_op.cc:341] FINISH: Updating metrics
0810 18:52:31.321369 (+ 74402us) write_op.cc:489] Releasing partition, row and schema locks
0810 18:52:31.323391 (+ 2022us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":279768,"reported_tablets":49,"child_traces":[["op",{"apply.queue_time_us":225226,"apply.run_cpu_time_us":18927,"apply.run_wall_time_us":1015035,"num_ops":30,"prepare.queue_time_us":94175,"prepare.run_cpu_time_us":33506,"prepare.run_wall_time_us":319980,"raft.queue_time_us":292804,"raft.run_cpu_time_us":79449,"raft.run_wall_time_us":178337,"replication_time_us":524997,"thread_start_us":476187,"threads_started":4,"wal-append.queue_time_us":125738,"wal-append.run_cpu_time_us":21652,"wal-append.run_wall_time_us":2568255}]]}
I20260810 18:52:39.261498 16695 leader_election.cc:302] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:39.262183 16632 raft_consensus.cc:1226] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 6. Current term is 7. Ops: []
W20260810 18:52:39.265563 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 74549.6us
I20260810 18:52:39.269188 21170 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:39.271238 21492 log.cc:924] Time spent T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.088s user 0.000s sys 0.001s
I20260810 18:52:39.264945 16624 raft_consensus.cc:2454] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 3.
W20260810 18:52:39.279951 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 101983us
I20260810 18:52:39.247509 21391 raft_consensus.cc:482] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:39.294365 21188 consensus_queue.cc:1046] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:52:39.296463 21469 consensus_queue.cc:1046] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:52:39.296451 16745 raft_consensus.cc:1226] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:39.300206 17724 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:52:39.265220 16701 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.997s (SENT) [suppressed 64 similar messages]
W20260810 18:52:39.303777 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 63874.4us
I20260810 18:52:39.304364 21299 raft_consensus.cc:3032] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 6 LEADER]: Stepping down as leader of term 6
I20260810 18:52:39.304723 21511 raft_consensus.cc:3032] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 LEADER]: Stepping down as leader of term 4
I20260810 18:52:39.305078 21444 consensus_queue.cc:1035] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:52:39.238615 21038 consensus_queue.cc:1035] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
W20260810 18:52:39.321563 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 56381.6us
I20260810 18:52:39.325379 17771 raft_consensus.cc:2454] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
W20260810 18:52:39.335879 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 54809us
I20260810 18:52:39.307740 21391 raft_consensus.cc:3037] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
W20260810 18:52:39.332831 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 177524us
I20260810 18:52:39.301481 21505 raft_consensus.cc:482] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:39.317055 21511 raft_consensus.cc:729] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:39.317196 21299 raft_consensus.cc:729] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:39.335511 21191 raft_consensus.cc:2781] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:39.347586 16448 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:39.299283 21359 consensus_queue.cc:1035] T c758b1e1139941a9a8c4e703cc058061 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.040s
I20260810 18:52:39.323414 21154 raft_consensus.cc:2781] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:39.299784 21460 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:39.335978 21503 consensus_queue.cc:1046] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20260810 18:52:39.337998 21493 raft_consensus.cc:3032] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 LEADER]: Stepping down as leader of term 6
I20260810 18:52:39.349709 21461 leader_election.cc:288] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:39.367972 16448 leader_election.cc:302] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:39.370898 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 103539us
I20260810 18:52:39.354332 21191 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:39.361097 21462 raft_consensus.cc:3032] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:39.348632 21505 raft_consensus.cc:504] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:39.338964 21439 consensus_queue.cc:1046] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260810 18:52:39.359256 21226 leader_election.cc:288] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:39.338429 21502 consensus_queue.cc:1035] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260810 18:52:39.362397 21493 raft_consensus.cc:729] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:39.380653 21462 raft_consensus.cc:729] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:39.399585 21498 consensus_queue.cc:1046] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
I20260810 18:52:39.400187 21462 consensus_queue.cc:260] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:39.400815 21513 consensus_queue.cc:1046] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
I20260810 18:52:39.401777 21462 raft_consensus.cc:3037] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:39.383368 21191 raft_consensus.cc:3037] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:39.394785 21296 raft_consensus.cc:2781] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Leader election won for term 9
I20260810 18:52:39.371942 21532 consensus_queue.cc:1046] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260810 18:52:39.398276 21271 consensus_queue.cc:1046] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.011s
I20260810 18:52:39.411271 21511 consensus_queue.cc:260] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:39.412976 16573 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:39.413141 21490 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:39.419220 21299 consensus_queue.cc:260] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:39.419766 21460 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:39.370405 21062 consensus_queue.cc:1046] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.030s
I20260810 18:52:39.418604 21191 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:39.421705 21505 leader_election.cc:288] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:39.422238 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 50628.7us
I20260810 18:52:39.362526 21434 consensus_queue.cc:1046] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
I20260810 18:52:39.465811 16622 raft_consensus.cc:1226] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:39.468550 16447 leader_election.cc:398] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Vote denied by peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387) with higher term. Message: Invalid argument: T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for earlier term 6. Current term is 7.
I20260810 18:52:39.433977 21359 raft_consensus.cc:2781] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260810 18:52:39.470729 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 102807us
W20260810 18:52:39.478083 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 110563us
W20260810 18:52:39.478866 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 65984.3us
I20260810 18:52:39.437309 21004 consensus_queue.cc:1046] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260810 18:52:39.468158 21154 raft_consensus.cc:686] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:39.486485 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:39.473387 21359 raft_consensus.cc:482] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:39.468487 21296 raft_consensus.cc:686] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [term 9 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:39.476305 21460 raft_consensus.cc:2781] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Leader election won for term 9
I20260810 18:52:39.467217 21437 consensus_queue.cc:1046] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.018s
W20260810 18:52:39.499673 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 61500.9us
I20260810 18:52:39.484035 21511 raft_consensus.cc:3037] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:39.502306 16745 raft_consensus.cc:2379] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 4: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:39.503252 16628 raft_consensus.cc:1226] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:39.505645 16447 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:39.494148 21272 raft_consensus.cc:3032] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:52:39.506515 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:39.506943 16633 raft_consensus.cc:1226] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:39.507455 21519 consensus_queue.cc:1046] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.106s
I20260810 18:52:39.519593 16573 leader_election.cc:302] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:39.494786 21359 raft_consensus.cc:3037] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:39.532337 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:39.533290 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:39.533622 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:39.533946 21154 consensus_queue.cc:237] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:39.535480 17775 raft_consensus.cc:1226] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260810 18:52:39.536888 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:39.513353 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:39.521368 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:39.522792 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 214749us
I20260810 18:52:39.523252 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:39.524819 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 171417us
I20260810 18:52:39.494482 21524 consensus_queue.cc:1035] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.160s
I20260810 18:52:39.525796 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:39.514317 21460 raft_consensus.cc:686] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 9 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:39.526544 21170 raft_consensus.cc:3032] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 7 LEADER]: Stepping down as leader of term 7
I20260810 18:52:39.526836 21191 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:39.499500 21062 consensus_queue.cc:1225] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer d304ac47fdf846de81337057a4ef7bea log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260810 18:52:39.528064 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:39.528659 21469 raft_consensus.cc:2781] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:39.530737 16750 raft_consensus.cc:1261] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260810 18:52:39.507481 21514 log.cc:924] Time spent T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.088s user 0.000s sys 0.001s
I20260810 18:52:39.545460 16450 leader_election.cc:302] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:39.546078 16636 raft_consensus.cc:1261] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260810 18:52:39.551288 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 50697us
I20260810 18:52:39.552999 21540 raft_consensus.cc:2726] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260810 18:52:39.515676 21272 raft_consensus.cc:729] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
W20260810 18:52:39.546952 16573 connection.cc:232] RPC call timeout handler was delayed by -1.8113810771610588s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1554s for a response
I20260810 18:52:39.494237 21507 consensus_queue.cc:1046] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.122s
I20260810 18:52:39.578768 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
W20260810 18:52:39.580032 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 66427.8us
I20260810 18:52:39.582609 21460 consensus_queue.cc:237] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:39.583649 21062 consensus_queue.cc:1035] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.005s
I20260810 18:52:39.590782 21507 raft_consensus.cc:3032] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:39.591322 21507 raft_consensus.cc:729] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:39.594383 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:39.547386 21170 raft_consensus.cc:729] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:39.559255 21391 raft_consensus.cc:504] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:39.584605 17724 leader_election.cc:398] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 election: Vote denied by peer d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791) with higher term. Message: Invalid argument: T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for earlier term 6. Current term is 7.
I20260810 18:52:39.597070 17786 raft_consensus.cc:2454] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
W20260810 18:52:39.591051 21527 log.cc:924] Time spent T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.096s user 0.001s sys 0.000s
I20260810 18:52:39.605093 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:39.606832 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:39.612480 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:39.487277 21493 consensus_queue.cc:260] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:39.617372 16749 raft_consensus.cc:2396] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260810 18:52:39.592941 21469 raft_consensus.cc:686] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:39.603823 21088 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:39.604696 21490 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:39.602905 21380 raft_consensus.cc:3032] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 4 LEADER]: Stepping down as leader of term 4
I20260810 18:52:39.638505 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:39.643122 16749 raft_consensus.cc:1261] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260810 18:52:39.645205 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:39.645550 16501 raft_consensus.cc:1226] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260810 18:52:39.646092 16506 raft_consensus.cc:2379] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 8: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:39.658116 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:39.658385 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:39.658635 16500 raft_consensus.cc:1226] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260810 18:52:39.659518 16695 leader_election.cc:302] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:39.660387 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:39.639717 21057 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:39.664095 17787 raft_consensus.cc:1226] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 9 LEADER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 8. Current term is 9. Ops: []
I20260810 18:52:39.664229 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:39.664917 21493 raft_consensus.cc:3037] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:39.664678 21057 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:39.667169 21057 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:39.666968 21469 consensus_queue.cc:237] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:39.667522 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:39.668313 21057 raft_consensus.cc:3032] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 9 LEADER]: Stepping down as leader of term 9
I20260810 18:52:39.664304 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:39.592399 21507 consensus_queue.cc:260] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:39.677440 21507 raft_consensus.cc:3037] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:39.641266 21380 raft_consensus.cc:729] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:39.680776 21531 consensus_queue.cc:1046] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:52:39.681427 21380 consensus_queue.cc:260] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:39.604264 21296 consensus_queue.cc:237] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:39.673058 21533 log.cc:924] Time spent T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.205s user 0.001s sys 0.000s
I20260810 18:52:39.637069 21088 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:39.688551 16502 raft_consensus.cc:1226] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:39.682610 21531 consensus_queue.cc:1046] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20260810 18:52:39.696691 21057 raft_consensus.cc:729] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:39.698215 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:39.686121 16447 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:39.688290 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
W20260810 18:52:39.698338 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 53978.8us
I20260810 18:52:39.723217 16501 raft_consensus.cc:2454] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
I20260810 18:52:39.734584 16496 raft_consensus.cc:2454] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 8.
I20260810 18:52:39.639854 21272 consensus_queue.cc:260] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:39.686007 21062 consensus_queue.cc:1035] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.038s
W20260810 18:52:39.741503 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 199573us
I20260810 18:52:39.742374 16450 leader_election.cc:302] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:39.711201 21380 raft_consensus.cc:3037] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
W20260810 18:52:39.754282 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 134014us
I20260810 18:52:39.724638 21565 raft_consensus.cc:3032] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 6 LEADER]: Stepping down as leader of term 6
I20260810 18:52:39.717721 21170 consensus_queue.cc:260] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:39.756587 21391 leader_election.cc:288] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:39.757009 21518 raft_consensus.cc:2726] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:39.757469 21571 raft_consensus.cc:3032] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Stepping down as leader of term 4
I20260810 18:52:39.758144 21570 raft_consensus.cc:2726] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:39.758013 21057 consensus_queue.cc:260] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:39.758525 16450 leader_election.cc:302] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:39.758805 21512 raft_consensus.cc:2726] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:39.759171 21537 raft_consensus.cc:2726] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:39.759505 21553 consensus_queue.cc:1046] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:52:39.759768 16695 leader_election.cc:302] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:39.760159 21191 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:39.761442 21299 raft_consensus.cc:3037] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
W20260810 18:52:39.776800 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 128058us
I20260810 18:52:39.751590 21558 consensus_queue.cc:1046] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.178s
I20260810 18:52:39.787025 21553 raft_consensus.cc:3032] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:52:39.798514 21553 raft_consensus.cc:729] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:39.799347 21553 consensus_queue.cc:260] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:39.800637 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:39.801313 16743 raft_consensus.cc:1226] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260810 18:52:39.801512 16742 raft_consensus.cc:2379] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 8: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:39.787279 21518 raft_consensus.cc:2781] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:39.802644 21518 raft_consensus.cc:482] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:39.803047 21518 raft_consensus.cc:3037] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:39.790952 16508 raft_consensus.cc:3037] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:39.807343 21088 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:39.808056 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 131656us
I20260810 18:52:39.811661 17724 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:39.812747 21308 raft_consensus.cc:3032] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 8 LEADER]: Stepping down as leader of term 8
I20260810 18:52:39.812641 21558 consensus_queue.cc:1046] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 18:52:39.813119 21308 raft_consensus.cc:729] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:39.794004 17775 raft_consensus.cc:1226] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:39.814014 21308 consensus_queue.cc:260] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:39.794854 16740 raft_consensus.cc:1226] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 LEADER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:39.815353 21535 consensus_queue.cc:1046] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260810 18:52:39.791833 21539 consensus_queue.cc:1035] T 42bb15fb54124b8db9b482e3f17fc87c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.024s
I20260810 18:52:39.795461 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:39.796657 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:39.764292 21565 raft_consensus.cc:729] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:39.797878 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:39.796154 17783 raft_consensus.cc:1261] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260810 18:52:39.794970 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:39.823277 16750 raft_consensus.cc:2379] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 5: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:39.817586 16748 raft_consensus.cc:2379] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 8: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:39.818931 16496 raft_consensus.cc:1261] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260810 18:52:39.820047 16501 raft_consensus.cc:1226] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:39.820767 16508 raft_consensus.cc:2454] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 7.
W20260810 18:52:39.833060 17726 leader_election.cc:334] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.589s (SENT)
I20260810 18:52:39.777647 21571 raft_consensus.cc:729] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:39.842823 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
W20260810 18:52:39.844472 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 86025.3us
I20260810 18:52:39.845435 16450 leader_election.cc:302] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:39.847437 21592 consensus_queue.cc:1046] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.011s
I20260810 18:52:39.800704 21553 raft_consensus.cc:3037] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:39.837559 17726 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:39.821123 21558 raft_consensus.cc:2726] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260810 18:52:39.831501 21170 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:39.831718 21308 raft_consensus.cc:3037] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:39.856853 21592 consensus_queue.cc:1225] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer 831925dd8b644ebab7eb0d672a94eddd log is divergent from this leader: its last log entry 9.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260810 18:52:39.861289 21557 consensus_queue.cc:1046] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260810 18:52:39.834025 21391 raft_consensus.cc:3032] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Stepping down as leader of term 6
I20260810 18:52:39.843642 21057 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Advancing to term 10
I20260810 18:52:39.787225 21191 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:39.878415 16450 leader_election.cc:302] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:39.857817 21088 raft_consensus.cc:2726] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260810 18:52:39.843957 21062 consensus_queue.cc:1035] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.032s
I20260810 18:52:39.834782 21513 consensus_queue.cc:1035] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.033s
I20260810 18:52:39.878454 21557 consensus_queue.cc:1225] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer d304ac47fdf846de81337057a4ef7bea log is divergent from this leader: its last log entry 7.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
W20260810 18:52:39.906862 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 74719.6us
I20260810 18:52:39.904512 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:39.914443 16626 raft_consensus.cc:2354] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for earlier term 8. Current term is 9.
I20260810 18:52:39.866649 21391 raft_consensus.cc:729] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:39.913031 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:39.921474 16623 raft_consensus.cc:1226] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [term 9 LEADER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 8. Current term is 9. Ops: []
I20260810 18:52:39.921839 16628 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 8.
I20260810 18:52:39.843112 21570 consensus_queue.cc:1035] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
I20260810 18:52:39.922525 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:39.879591 21188 raft_consensus.cc:3032] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Stepping down as leader of term 3
W20260810 18:52:39.923260 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 90224.5us
I20260810 18:52:39.875558 21469 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:39.857514 21540 consensus_queue.cc:1046] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260810 18:52:39.935202 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:39.930456 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:39.933765 16573 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:39.937439 16628 raft_consensus.cc:2379] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 7: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:39.870718 21359 raft_consensus.cc:504] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:52:39.938337 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:39.938750 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 60419.4us
I20260810 18:52:39.908164 21460 raft_consensus.cc:482] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:39.939838 16450 leader_election.cc:302] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:39.923990 21502 raft_consensus.cc:482] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:39.914111 21577 raft_consensus.cc:2726] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:52:39.930197 21568 raft_consensus.cc:2726] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:39.943766 16622 raft_consensus.cc:1261] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260810 18:52:39.943054 16448 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:39.953095 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:39.922609 21490 raft_consensus.cc:482] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:40.025298 16633 raft_consensus.cc:2454] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
I20260810 18:52:39.931169 21188 raft_consensus.cc:729] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:39.858349 21565 consensus_queue.cc:260] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:40.024503 21491 log.cc:924] Time spent T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20260810 18:52:40.051967 16631 raft_consensus.cc:1203] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Deduplicated request from leader. Original: 5.6->[6.7-6.7] Dedup: 6.7->[]
I20260810 18:52:40.055970 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:40.057217 16631 raft_consensus.cc:1226] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 6. Current term is 7. Ops: [6.7-6.7]
W20260810 18:52:40.052459 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 115249us
I20260810 18:52:40.061647 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:40.036777 21088 raft_consensus.cc:2726] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260810 18:52:40.066622 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 126904us
I20260810 18:52:40.067847 16450 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:40.068131 17787 raft_consensus.cc:3032] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 6 LEADER]: Stepping down as leader of term 6
I20260810 18:52:40.068524 17787 raft_consensus.cc:729] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:40.074787 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:40.075429 16749 raft_consensus.cc:3037] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:40.081012 16504 raft_consensus.cc:2454] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 7.
I20260810 18:52:40.089648 14467 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260810 18:52:40.092414 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 163730us
I20260810 18:52:40.093935 16622 raft_consensus.cc:1226] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:39.953039 21469 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:39.883836 21571 consensus_queue.cc:260] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:40.015368 21460 raft_consensus.cc:504] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:40.116230 16626 raft_consensus.cc:1226] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 2. Current term is 3. Ops: []
I20260810 18:52:40.084043 21537 raft_consensus.cc:2781] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260810 18:52:40.136960 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:40.138200 16622 raft_consensus.cc:2379] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 7: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:40.123426 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:40.124311 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:40.036439 21502 raft_consensus.cc:504] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:40.142678 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:40.106686 21565 raft_consensus.cc:3037] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:40.089072 21546 raft_consensus.cc:2726] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:40.044094 21490 raft_consensus.cc:504] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:40.144728 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:40.135738 17787 consensus_queue.cc:260] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:40.168056 16626 raft_consensus.cc:2362] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 7. Re-sending same reply.
I20260810 18:52:40.156270 16501 raft_consensus.cc:1226] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260810 18:52:40.118155 21532 raft_consensus.cc:2726] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260810 18:52:40.179605 16505 raft_consensus.cc:1226] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260810 18:52:40.183488 17787 raft_consensus.cc:3037] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:40.184058 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
W20260810 18:52:40.183729 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 98101.5us
I20260810 18:52:40.135844 21537 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:40.185668 16450 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:40.050953 21391 consensus_queue.cc:260] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:40.144016 21540 raft_consensus.cc:482] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:40.194710 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 102919us
I20260810 18:52:40.192250 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:40.147027 21498 raft_consensus.cc:482] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:40.196729 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:40.201121 16573 leader_election.cc:302] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:40.201560 21602 log.cc:924] Time spent T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20260810 18:52:40.201927 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:40.213382 16631 raft_consensus.cc:2396] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260810 18:52:40.204568 16573 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:40.208339 17787 raft_consensus.cc:1261] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260810 18:52:40.167982 21191 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:40.208982 16450 leader_election.cc:302] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:40.186580 21546 raft_consensus.cc:482] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:40.219743 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:40.195619 21537 raft_consensus.cc:3037] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Advancing to term 9
W20260810 18:52:40.208668 21572 log.cc:924] Time spent T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.154s user 0.000s sys 0.001s
I20260810 18:52:40.111420 21188 consensus_queue.cc:260] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:40.236843 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:40.155416 21495 consensus_queue.cc:1046] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20260810 18:52:40.214114 21493 raft_consensus.cc:2781] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:40.201947 21592 raft_consensus.cc:2781] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election won for term 7
I20260810 18:52:40.203194 21526 raft_consensus.cc:2726] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260810 18:52:40.259547 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:40.252028 16745 raft_consensus.cc:3037] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:40.217360 16448 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:40.224103 21554 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260810 18:52:40.264354 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:40.231820 21391 raft_consensus.cc:3037] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:40.236280 21549 raft_consensus.cc:3032] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 6 LEADER]: Stepping down as leader of term 6
I20260810 18:52:40.240491 21519 raft_consensus.cc:2781] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader election won for term 8
I20260810 18:52:40.268003 16628 raft_consensus.cc:2379] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 7: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:40.269155 16450 leader_election.cc:302] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:40.276610 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:40.271150 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:40.271409 16636 raft_consensus.cc:1226] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 8. Current term is 9. Ops: []
I20260810 18:52:40.274179 16505 raft_consensus.cc:1261] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260810 18:52:40.277652 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:40.258391 21570 raft_consensus.cc:2781] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:40.232040 21271 consensus_queue.cc:1035] T 6a1d591f794a4601bf3c950c2c57b3cf P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20260810 18:52:40.215490 21498 raft_consensus.cc:504] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:40.267863 21188 raft_consensus.cc:3037] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:40.265625 21359 leader_election.cc:288] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:40.223436 21540 raft_consensus.cc:504] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:40.255896 21004 consensus_queue.cc:1035] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.029s
I20260810 18:52:40.304646 16622 raft_consensus.cc:1226] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:40.298286 16740 raft_consensus.cc:3037] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:40.280318 21524 raft_consensus.cc:3032] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 4 LEADER]: Stepping down as leader of term 4
I20260810 18:52:40.274793 21493 raft_consensus.cc:686] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:40.310256 21598 raft_consensus.cc:482] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:40.310714 21608 raft_consensus.cc:482] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:40.329061 21608 raft_consensus.cc:504] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:40.329593 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:40.270608 21532 consensus_queue.cc:1035] T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260810 18:52:40.339548 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:40.320322 21271 raft_consensus.cc:2781] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election won for term 7
W20260810 18:52:40.320749 21594 log.cc:924] Time spent T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.330s user 0.000s sys 0.001s
I20260810 18:52:40.301789 21226 consensus_queue.cc:1035] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.020s
I20260810 18:52:40.321537 21613 raft_consensus.cc:2781] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260810 18:52:40.322410 16631 raft_consensus.cc:1226] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:40.275760 21549 raft_consensus.cc:729] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:40.314287 21439 raft_consensus.cc:3032] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 8 LEADER]: Stepping down as leader of term 8
I20260810 18:52:40.252971 21546 raft_consensus.cc:504] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:40.332803 16749 raft_consensus.cc:2454] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 9.
I20260810 18:52:40.337615 21540 leader_election.cc:288] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:40.340250 16505 raft_consensus.cc:2454] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 7.
I20260810 18:52:40.340421 16745 raft_consensus.cc:2454] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:40.305282 21519 raft_consensus.cc:686] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:40.302390 21570 raft_consensus.cc:686] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 6 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:40.381938 16573 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:40.383162 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:52:40.383466 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 166212us
I20260810 18:52:40.384188 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:40.417447 16633 raft_consensus.cc:3037] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:40.354070 21495 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:40.359313 21518 consensus_meta.cc:287] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.553s user 0.009s sys 0.002s
I20260810 18:52:40.387379 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:40.324141 21524 raft_consensus.cc:729] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
W20260810 18:52:40.406035 21587 log.cc:924] Time spent T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.310s user 0.001s sys 0.000s
I20260810 18:52:40.411410 21493 consensus_queue.cc:237] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:40.410739 16740 raft_consensus.cc:2454] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 7.
I20260810 18:52:40.381112 21469 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:40.412963 21546 leader_election.cc:288] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:40.405848 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:40.429586 21524 consensus_queue.cc:260] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:40.420162 21170 raft_consensus.cc:2781] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:40.430771 21170 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:40.404054 21271 raft_consensus.cc:686] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:40.431026 21524 raft_consensus.cc:3037] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:40.431932 21271 consensus_queue.cc:237] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:40.435763 16701 leader_election.cc:334] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.542s (SENT)
I20260810 18:52:40.436395 16701 leader_election.cc:302] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
W20260810 18:52:40.435983 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 463793us
I20260810 18:52:40.359829 21439 raft_consensus.cc:729] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:40.360011 21613 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260810 18:52:40.419909 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 477091us
I20260810 18:52:40.429589 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:40.438508 21170 raft_consensus.cc:3037] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:40.426191 21518 raft_consensus.cc:504] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:40.444489 21543 raft_consensus.cc:3032] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 6 LEADER]: Stepping down as leader of term 6
I20260810 18:52:40.444918 21543 raft_consensus.cc:729] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:40.446022 21518 leader_election.cc:288] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:40.445715 21543 consensus_queue.cc:260] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:40.447357 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:40.447947 16622 raft_consensus.cc:3037] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:40.448333 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:40.449013 21613 raft_consensus.cc:3037] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:40.447791 21439 consensus_queue.cc:260] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:40.451292 21439 raft_consensus.cc:3037] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Advancing to term 9
W20260810 18:52:40.455796 21380 consensus_meta.cc:287] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.691s user 0.011s sys 0.000s
I20260810 18:52:40.456827 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:40.458267 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:40.446702 21537 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:40.459374 21543 raft_consensus.cc:3037] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:40.461126 21537 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:40.463841 16631 raft_consensus.cc:2379] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 7: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:40.415458 21502 leader_election.cc:288] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:40.419459 21460 leader_election.cc:288] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:40.465323 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:40.391975 17724 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.014s sys 0.014s
W20260810 18:52:40.468530 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 199459us
W20260810 18:52:40.469161 17724 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.601s user 0.027s sys 0.014s
I20260810 18:52:40.470125 16505 raft_consensus.cc:2454] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:52:40.341640 21598 raft_consensus.cc:504] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:40.443259 21507 consensus_meta.cc:287] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.763s user 0.012s sys 0.002s
W20260810 18:52:40.471254 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 616496us
I20260810 18:52:40.472255 16573 leader_election.cc:302] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:40.426681 21495 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:40.439694 21549 consensus_queue.cc:260] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:40.672796 16622 raft_consensus.cc:2454] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:40.539671 16633 raft_consensus.cc:2396] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260810 18:52:40.564913 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:52:40.475737 17727 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.593s user 0.010s sys 0.001s
W20260810 18:52:40.670800 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 198611us
W20260810 18:52:40.674084 17727 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.791s user 0.015s sys 0.011s
W20260810 18:52:40.524596 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 88881.4us
I20260810 18:52:40.675151 16631 raft_consensus.cc:2396] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260810 18:52:40.642299 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:40.674469 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 791499us
I20260810 18:52:40.677277 17771 raft_consensus.cc:1203] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 9 LEADER]: Deduplicated request from leader. Original: 6.8->[8.9-8.9] Dedup: 8.9->[]
I20260810 18:52:40.677778 17771 raft_consensus.cc:1226] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 9 LEADER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260810 18:52:40.494663 21188 consensus_queue.cc:1035] T dcd14258d5d24692a3377387cb5c204e P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.184s
I20260810 18:52:40.468181 21608 leader_election.cc:288] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:40.681021 16701 leader_election.cc:334] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.638s (SENT)
I20260810 18:52:40.479914 21537 consensus_queue.cc:1035] T da1fa324d17c479383225a8cea494a80 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.201s
I20260810 18:52:40.681689 21380 raft_consensus.cc:482] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:40.682914 21380 raft_consensus.cc:504] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:40.683984 16701 leader_election.cc:334] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.671s (SENT)
W20260810 18:52:40.684173 21057 consensus_meta.cc:287] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.806s user 0.009s sys 0.001s
I20260810 18:52:40.684386 16701 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:40.684967 21380 leader_election.cc:288] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:40.681794 21461 raft_consensus.cc:482] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:40.680292 21553 consensus_meta.cc:287] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.822s user 0.010s sys 0.002s
I20260810 18:52:40.688400 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:40.690446 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:40.689339 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:40.686964 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:40.690753 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:40.693616 16505 raft_consensus.cc:2454] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 7.
I20260810 18:52:40.693904 16631 raft_consensus.cc:2379] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 8: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:40.523298 16501 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b"
dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
mode: GRACEFUL
new_leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
from {username='slave'} at 127.0.0.1:51572
I20260810 18:52:40.682396 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:52:40.644662 21642 log.cc:924] Time spent T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260810 18:52:40.696056 21532 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:40.696563 16747 raft_consensus.cc:1226] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260810 18:52:40.696593 21532 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:40.699208 21532 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:40.702986 21495 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:40.703285 21603 raft_consensus.cc:2726] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260810 18:52:40.703680 17723 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.621s user 0.004s sys 0.015s
W20260810 18:52:40.704034 17723 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.759s user 0.004s sys 0.015s
I20260810 18:52:40.705883 21630 consensus_queue.cc:1046] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 9, Time since last communication: 0.001s
I20260810 18:52:40.706822 21630 consensus_queue.cc:1225] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer 831925dd8b644ebab7eb0d672a94eddd log is divergent from this leader: its last log entry 9.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260810 18:52:40.694474 16501 raft_consensus.cc:595] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [term 6 LEADER]: Received request to transfer leadership to TS 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:52:40.707691 21617 log.cc:924] Time spent T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.349s user 0.001s sys 0.000s
W20260810 18:52:40.523313 21556 log.cc:924] Time spent T 00000000000000000000000000000000 P 0975c6fa18274f2baee7fb3615d5b985: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20260810 18:52:40.716236 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:40.469165 21546 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:40.711790 16622 raft_consensus.cc:2362] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 8. Re-sending same reply.
I20260810 18:52:40.712965 16573 leader_election.cc:302] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:40.692898 16623 raft_consensus.cc:2362] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4. Re-sending same reply.
I20260810 18:52:40.714186 16633 raft_consensus.cc:2379] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 7: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:40.685223 21608 raft_consensus.cc:2726] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260810 18:52:40.719264 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:40.675866 21460 raft_consensus.cc:2781] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:40.583701 21518 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:40.722362 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:40.723718 16501 raft_consensus.cc:1261] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260810 18:52:40.724154 16623 raft_consensus.cc:3037] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:40.722579 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:40.725621 16701 leader_election.cc:334] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.673s (SENT)
I20260810 18:52:40.726457 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:40.722368 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:40.728394 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:40.732488 17785 raft_consensus.cc:2379] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 7: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:52:40.733263 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:40.733948 16631 raft_consensus.cc:3037] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:40.736362 16701 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:40.733880 16622 raft_consensus.cc:2454] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:40.737419 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:40.723881 21495 raft_consensus.cc:2781] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:40.744606 16740 raft_consensus.cc:2454] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 5.
W20260810 18:52:40.745594 16701 leader_election.cc:334] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.731s (SENT)
W20260810 18:52:40.745774 21618 log.cc:924] Time spent T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.332s user 0.001s sys 0.000s
I20260810 18:52:40.746110 21608 raft_consensus.cc:2726] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:40.746716 21598 leader_election.cc:288] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:40.749156 21565 consensus_meta.cc:287] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.581s user 0.008s sys 0.001s
I20260810 18:52:40.723016 21461 raft_consensus.cc:504] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:40.735107 21498 leader_election.cc:288] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:40.754558 21170 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:40.746884 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:40.763154 16631 raft_consensus.cc:2454] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:40.771737 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:40.773144 16622 raft_consensus.cc:2379] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 7: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:40.725450 21460 raft_consensus.cc:482] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:40.762159 16701 leader_election.cc:302] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:40.771831 16505 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7dd134517a994065a45633249a3ce405"
dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
mode: GRACEFUL
new_leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
from {username='slave'} at 127.0.0.1:51572
I20260810 18:52:40.777230 16505 raft_consensus.cc:595] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Received request to transfer leadership to TS 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:40.601253 21519 consensus_queue.cc:237] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:40.764693 21495 raft_consensus.cc:482] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:40.774358 16448 leader_election.cc:302] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:40.765902 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:40.730509 21359 consensus_queue.cc:1046] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
I20260810 18:52:40.568511 21570 consensus_queue.cc:237] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:40.787515 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:40.794312 16695 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:40.724707 21546 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:40.736662 21518 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:40.801394 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:40.803048 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:40.803308 16622 raft_consensus.cc:2379] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 8: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:40.802471 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:40.804723 16633 raft_consensus.cc:3037] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 5
I20260810 18:52:40.804847 16632 raft_consensus.cc:2396] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260810 18:52:40.779655 21460 raft_consensus.cc:3037] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:40.765961 21502 consensus_queue.cc:1035] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
I20260810 18:52:40.810709 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:40.813361 16740 raft_consensus.cc:2454] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:40.790187 21495 raft_consensus.cc:3037] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:40.815160 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:40.782105 21498 raft_consensus.cc:482] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:40.823997 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 78450.6us
W20260810 18:52:40.825736 16701 leader_election.cc:334] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.737s (SENT)
I20260810 18:52:40.825197 16506 raft_consensus.cc:2454] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 7.
I20260810 18:52:40.835536 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:40.837448 16633 raft_consensus.cc:2454] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:40.802117 21532 raft_consensus.cc:2781] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:40.823320 16747 raft_consensus.cc:2454] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 4.
I20260810 18:52:40.836413 16632 raft_consensus.cc:2454] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:40.802034 21608 raft_consensus.cc:2726] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:52:40.850235 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:40.851115 16633 raft_consensus.cc:2379] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 7: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:40.824694 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:40.852866 16740 raft_consensus.cc:1226] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:40.813067 21598 raft_consensus.cc:2781] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader election won for term 8
I20260810 18:52:40.788924 21565 consensus_queue.cc:1035] T 0a481385212f4e189ea5b2f5e215be4d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20260810 18:52:40.854823 16505 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43"
dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
mode: GRACEFUL
new_leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
from {username='slave'} at 127.0.0.1:51572
W20260810 18:52:40.810693 16323 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 8810b6b0b676405daf5ffddc50a4d21f with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica d304ac47fdf846de81337057a4ef7bea is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:52:40.856644 21532 raft_consensus.cc:686] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
W20260810 18:52:40.859501 16323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 51929.8us
I20260810 18:52:40.861127 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:40.862665 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:40.862866 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:52:40.862030 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 91377.5us
I20260810 18:52:40.867540 17727 leader_election.cc:302] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:40.872947 16748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 448) took 1033 ms. Trace:
W20260810 18:52:40.860481 21308 consensus_meta.cc:287] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.992s user 0.012s sys 0.001s
I20260810 18:52:40.873492 16748 rpcz_store.cc:276] 0810 18:52:39.832722 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:39.832869 (+ 147us) service_pool.cc:227] Handling call
0810 18:52:39.848564 (+ 15695us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:40.066868 (+218304us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:40.066904 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:40.067378 (+ 474us) log.cc:842] Serialized 26 byte log entry
0810 18:52:40.256272 (+188894us) raft_consensus.cc:1631] Marking committed up to 7
0810 18:52:40.256326 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:40.256418 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:40.865872 (+609454us) raft_consensus.cc:1671] finished
0810 18:52:40.865900 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:40.866048 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":188302,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":13572,"thread_start_us":331360,"threads_started":2,"wal-append.queue_time_us":200500}
I20260810 18:52:40.880383 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:40.885932 16740 raft_consensus.cc:2379] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 7: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:40.886261 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:40.887105 16495 raft_consensus.cc:3037] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:40.888221 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:40.889998 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 64291.8us
I20260810 18:52:40.861769 21188 raft_consensus.cc:482] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:40.893152 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:40.894567 16745 raft_consensus.cc:1226] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 3. Current term is 5. Ops: []
I20260810 18:52:40.896466 17786 raft_consensus.cc:1261] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20260810 18:52:40.898780 17786 pending_rounds.cc:76] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd: Aborting all ops after (but not including) 7
I20260810 18:52:40.899257 17786 pending_rounds.cc:97] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260810 18:52:40.899578 17786 raft_consensus.cc:2866] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:52:40.846388 21498 raft_consensus.cc:504] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:40.906107 21498 leader_election.cc:288] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:40.908098 16633 raft_consensus.cc:2379] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 6: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:40.906888 16747 raft_consensus.cc:1261] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260810 18:52:40.903208 16701 leader_election.cc:334] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.504s (SENT)
I20260810 18:52:40.909543 16701 leader_election.cc:302] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:40.904232 17775 raft_consensus.cc:1261] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260810 18:52:40.880048 21038 raft_consensus.cc:482] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:40.932349 16495 raft_consensus.cc:2454] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:40.954519 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:40.910851 21592 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.612s user 0.025s sys 0.032s
W20260810 18:52:40.955245 17726 leader_election.cc:334] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.183s (SENT)
I20260810 18:52:40.946874 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:40.955718 17726 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:40.891789 21598 raft_consensus.cc:686] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
W20260810 18:52:40.964847 21592 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.677s user 0.026s sys 0.032s
I20260810 18:52:40.994683 16747 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 8.
I20260810 18:52:41.149338 16506 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52338 (request call id 1612) took 1013 ms. Trace:
I20260810 18:52:41.104658 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:41.154024 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:52:41.196040 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 291797us
I20260810 18:52:41.103264 16745 raft_consensus.cc:1261] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260810 18:52:41.102092 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 198930us
I20260810 18:52:41.205665 16506 rpcz_store.cc:276] 0810 18:52:40.136348 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:40.136598 (+ 250us) service_pool.cc:227] Handling call
0810 18:52:41.149133 (+1012535us) spinlock_profiling.cc:243] Waited 622 ms on lock 0x7b54001ff9d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:41.149270 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1430272896}
I20260810 18:52:41.095561 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:41.138983 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 227229us
I20260810 18:52:41.034112 16622 raft_consensus.cc:2454] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
W20260810 18:52:41.209750 16701 leader_election.cc:334] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.787s (SENT)
I20260810 18:52:41.140170 17770 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55586 (request call id 457) took 1347 ms. Trace:
I20260810 18:52:41.158048 16636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60066 (request call id 328) took 1024 ms. Trace:
I20260810 18:52:41.101945 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:40.913348 21608 raft_consensus.cc:482] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:41.207526 16501 raft_consensus.cc:2362] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5. Re-sending same reply.
I20260810 18:52:41.210246 16701 leader_election.cc:302] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:41.203451 16623 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 9.
I20260810 18:52:41.203539 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:41.211951 16632 raft_consensus.cc:3037] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:40.994346 21592 raft_consensus.cc:686] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:41.215993 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:41.216709 16747 raft_consensus.cc:2454] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 8.
I20260810 18:52:41.216015 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:41.217701 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:41.210474 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:41.214994 16448 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:41.156767 21391 consensus_meta.cc:287] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.881s user 0.009s sys 0.001s
W20260810 18:52:41.139304 21439 consensus_meta.cc:287] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.688s user 0.006s sys 0.005s
I20260810 18:52:41.218298 16506 raft_consensus.cc:3037] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:41.219988 16633 raft_consensus.cc:2454] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:41.221696 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:41.151866 21620 log.cc:924] Time spent T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.887s user 0.005s sys 0.001s
W20260810 18:52:41.125373 21649 log.cc:924] Time spent T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.282s user 0.001s sys 0.000s
W20260810 18:52:41.125871 21652 log.cc:924] Time spent T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20260810 18:52:41.210522 17770 rpcz_store.cc:276] 0810 18:52:39.792744 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:39.792857 (+ 113us) service_pool.cc:227] Handling call
0810 18:52:39.793218 (+ 361us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:39.920645 (+127427us) raft_consensus.cc:1520] Early marking committed up to index 7
0810 18:52:40.129559 (+208914us) log.cc:842] Serialized 14 byte log entry
0810 18:52:40.273650 (+144091us) log.cc:842] Serialized 14 byte log entry
0810 18:52:40.286568 (+ 12918us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:40.286617 (+ 49us) raft_consensus.cc:1631] Marking committed up to 7
0810 18:52:40.286667 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:41.139270 (+852603us) spinlock_profiling.cc:243] Waited 7.81 ms on lock 0x7b18004334e0. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1aee0718b1 00007f1aee07e661 00007f1ae519ff96 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9
0810 18:52:41.139883 (+ 613us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b54005401d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:41.139938 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:41.140095 (+ 157us) inbound_call.cc:173] Queueing success response
0810 18:52:41.161857 (+ 21762us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x7b18004334e0. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1aee0718b1 00007f1aee07e661 00007f1ae51d2764 00007f1ae51d5b11 00007f1ae51d5ac9 00007f1ae51d5a59 00007f1ae51d5a21 00007f1ae51d4d1d 00007f1aee436204 00007f1aee436039 00007f1ae51b4dc6
Metrics: {"prepare.queue_time_us":128959,"prepare.run_cpu_time_us":137,"prepare.run_wall_time_us":11797,"raft.queue_time_us":165835,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":35768,"spinlock_wait_cycles":318762880,"thread_start_us":339842,"threads_started":2,"wal-append.queue_time_us":206249}
I20260810 18:52:40.909225 21188 raft_consensus.cc:504] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:41.210713 16636 rpcz_store.cc:276] 0810 18:52:40.133959 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:40.134125 (+ 166us) service_pool.cc:227] Handling call
0810 18:52:40.134590 (+ 465us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:40.285460 (+150870us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:41.157799 (+872339us) spinlock_profiling.cc:243] Waited 134 ms on lock 0x7b54003859d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:41.157984 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":307112192}
I20260810 18:52:40.997593 21644 raft_consensus.cc:482] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:40.999121 21461 leader_election.cc:288] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:41.222265 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:41.230564 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:41.231204 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:41.057924 21502 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:40.954543 21603 raft_consensus.cc:2726] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260810 18:52:41.252096 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:41.231775 16701 leader_election.cc:334] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.565s (SENT)
I20260810 18:52:41.207320 21568 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:41.160985 21272 raft_consensus.cc:3037] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:41.123317 21543 raft_consensus.cc:2726] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260810 18:52:41.161016 21546 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:41.131206 21518 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:41.007193 21170 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:41.082567 21640 raft_consensus.cc:482] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:41.233952 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:41.131233 21444 raft_consensus.cc:3032] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 8 LEADER]: Stepping down as leader of term 8
I20260810 18:52:41.077757 21524 raft_consensus.cc:482] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:41.243172 21493 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.575s user 0.001s sys 0.015s
I20260810 18:52:41.115654 21299 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:40.951346 21308 raft_consensus.cc:2781] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:41.019167 21498 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:41.226981 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:41.238234 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:41.273514 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:41.274158 16749 raft_consensus.cc:3037] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:41.246357 17783 raft_consensus.cc:1261] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260810 18:52:41.246718 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:41.222146 21577 raft_consensus.cc:2781] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Leader election won for term 9
I20260810 18:52:41.216015 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:41.150926 21004 consensus_queue.cc:1035] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.091s
I20260810 18:52:41.084026 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:41.255208 16636 raft_consensus.cc:2379] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 10: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:41.255712 16750 raft_consensus.cc:1226] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 8. Current term is 9. Ops: []
I20260810 18:52:41.135766 21555 consensus_queue.cc:1035] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.106s
I20260810 18:52:40.963028 21038 raft_consensus.cc:504] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:41.258096 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:41.258208 16701 leader_election.cc:302] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:41.236858 21154 raft_consensus.cc:2726] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260810 18:52:41.236702 21539 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:41.077565 21380 raft_consensus.cc:482] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:41.006616 21532 consensus_queue.cc:237] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:52:41.247241 21461 raft_consensus.cc:482] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:41.274201 16622 raft_consensus.cc:2379] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 5: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:41.241609 16695 leader_election.cc:302] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:41.271842 21444 raft_consensus.cc:729] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:41.269425 16632 raft_consensus.cc:2454] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:41.277323 16628 raft_consensus.cc:2396] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260810 18:52:41.270159 16506 raft_consensus.cc:2454] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:41.222990 21608 raft_consensus.cc:504] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:41.284725 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 75919.5us
I20260810 18:52:41.304497 16749 raft_consensus.cc:2454] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
W20260810 18:52:41.276156 21493 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.608s user 0.002s sys 0.016s
I20260810 18:52:41.126446 21598 consensus_queue.cc:237] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:41.268908 21654 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:41.330976 16575 leader_election.cc:302] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:41.275770 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 9 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:41.388612 16633 raft_consensus.cc:2379] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 9: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:41.274714 21308 raft_consensus.cc:482] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:41.333978 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:41.334826 21461 raft_consensus.cc:504] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:41.335515 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 111397us
I20260810 18:52:41.392205 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:41.393134 16750 raft_consensus.cc:2379] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 5: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:41.336014 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:41.336156 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:41.341428 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 99922.6us
I20260810 18:52:41.395807 16695 leader_election.cc:302] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:41.342093 21154 raft_consensus.cc:2726] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:41.349555 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:41.343890 21553 raft_consensus.cc:482] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:41.309331 21577 raft_consensus.cc:686] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 9 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:41.270632 21568 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:41.385002 21519 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:41.343377 21667 raft_consensus.cc:482] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:41.419462 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:41.419888 21667 raft_consensus.cc:504] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:41.420421 17786 raft_consensus.cc:2454] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 7.
W20260810 18:52:41.422021 16508 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52338 (request call id 1608) took 1779 ms (client timeout 2170 ms). Trace:
I20260810 18:52:41.386140 16505 raft_consensus.cc:595] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Received request to transfer leadership to TS 0397d760589d40d3b92ef141cdfc2dbb
W20260810 18:52:41.333211 21613 consensus_meta.cc:287] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.883s user 0.009s sys 0.003s
I20260810 18:52:41.423447 21613 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:41.425674 21613 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:41.258332 21502 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:41.426836 21038 leader_election.cc:288] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:41.427527 16695 leader_election.cc:302] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:41.244745 21644 raft_consensus.cc:504] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:41.395826 16500 raft_consensus.cc:2379] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 8: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
W20260810 18:52:41.395005 16701 leader_election.cc:334] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.650s (SENT)
I20260810 18:52:41.396876 16633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1687) took 1074 ms. Trace:
I20260810 18:52:41.399525 16506 raft_consensus.cc:3037] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:41.356593 16636 raft_consensus.cc:1226] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 4. Current term is 6. Ops: [4.3-4.3]
I20260810 18:52:41.399780 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:41.277655 21524 raft_consensus.cc:504] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:41.393325 17726 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:41.331737 21649 log.cc:924] Time spent T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260810 18:52:41.405110 17787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55570 (request call id 548) took 1730 ms. Trace:
I20260810 18:52:41.279138 21498 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:41.357474 21513 raft_consensus.cc:2726] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:52:41.274341 21640 raft_consensus.cc:504] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:41.278682 21299 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:41.411747 16628 raft_consensus.cc:1226] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 6. Current term is 7. Ops: []
W20260810 18:52:41.412739 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 81843.2us
I20260810 18:52:41.349983 21380 raft_consensus.cc:504] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:41.412277 21170 raft_consensus.cc:2726] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260810 18:52:41.417686 21593 log.cc:924] Time spent T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 1.332s user 0.004s sys 0.000s
I20260810 18:52:41.418040 21568 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:41.333818 21539 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:41.418777 21519 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:41.331645 21495 raft_consensus.cc:504] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:41.422756 21667 leader_election.cc:288] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:41.398057 21439 consensus_queue.cc:1046] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.079s
I20260810 18:52:41.397454 21437 consensus_queue.cc:1046] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.086s
I20260810 18:52:41.334015 21592 consensus_queue.cc:237] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:41.331378 17780 raft_consensus.cc:2454] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 7.
I20260810 18:52:41.397974 21308 raft_consensus.cc:3037] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:41.773042 16631 raft_consensus.cc:2354] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 7. Current term is 8.
I20260810 18:52:41.782176 16575 leader_election.cc:302] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:41.782716 16573 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:41.781656 16628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 2139) took 1334 ms. Trace:
I20260810 18:52:41.783442 16633 rpcz_store.cc:276] 0810 18:52:40.322939 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:41.275473 (+952534us) service_pool.cc:227] Handling call
0810 18:52:41.396807 (+121334us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:41.529772 21188 leader_election.cc:288] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:41.784651 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:41.784775 21535 raft_consensus.cc:2781] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:41.785324 16633 raft_consensus.cc:3037] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:41.472321 16623 raft_consensus.cc:2396] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260810 18:52:41.786402 17787 rpcz_store.cc:276] 0810 18:52:39.674851 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:39.674992 (+ 141us) service_pool.cc:227] Handling call
0810 18:52:39.692662 (+ 17670us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:40.214177 (+521515us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:41.404789 (+1190612us) spinlock_profiling.cc:243] Waited 375 ms on lock 0x7b540062ff54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:41.405027 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":863180032}
I20260810 18:52:41.788885 21524 leader_election.cc:288] T 6a1d591f794a4601bf3c950c2c57b3cf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:41.790037 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:41.790704 21154 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:41.791254 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:41.523693 21673 log.cc:924] Time spent T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.259s user 0.001s sys 0.000s
I20260810 18:52:41.791960 16501 raft_consensus.cc:2454] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:52:41.791249 21154 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:41.540095 21618 log.cc:924] Time spent T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.111s user 0.000s sys 0.001s
W20260810 18:52:41.422569 16508 rpcz_store.cc:269] 0810 18:52:39.644562 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:39.648466 (+ 3904us) service_pool.cc:227] Handling call
0810 18:52:41.421815 (+1773349us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b54001379d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:41.421946 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":262869888}
I20260810 18:52:41.794095 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:41.794670 21498 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:41.795050 16508 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 8.
W20260810 18:52:41.491802 21460 consensus_meta.cc:287] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.669s user 0.011s sys 0.000s
I20260810 18:52:41.535391 21577 consensus_queue.cc:237] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:41.795970 21460 raft_consensus.cc:504] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:41.476749 21226 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:41.797170 21226 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:41.504717 21518 raft_consensus.cc:2781] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:41.799391 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:41.425686 21444 consensus_queue.cc:260] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:41.799718 21462 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.004s sys 0.016s
I20260810 18:52:41.785471 21535 raft_consensus.cc:686] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:41.806960 16495 raft_consensus.cc:2362] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 9. Re-sending same reply.
I20260810 18:52:41.428064 21502 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:41.798563 21460 leader_election.cc:288] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:41.512040 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 84587.5us
I20260810 18:52:41.816268 16695 leader_election.cc:302] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:41.817474 16632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 2137) took 1370 ms. Trace:
I20260810 18:52:41.817837 16632 rpcz_store.cc:276] 0810 18:52:40.446672 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:41.346816 (+900144us) service_pool.cc:227] Handling call
0810 18:52:41.347263 (+ 447us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:41.347560 (+ 297us) raft_consensus.cc:1520] Early marking committed up to index 4
0810 18:52:41.748674 (+401114us) log.cc:842] Serialized 14 byte log entry
0810 18:52:41.817170 (+ 68496us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:41.817219 (+ 49us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:41.817258 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:41.817320 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:41.817421 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2779,"prepare.queue_time_us":424926,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":43,"thread_start_us":414882,"threads_started":2}
I20260810 18:52:41.692829 21512 raft_consensus.cc:482] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:41.784312 16628 rpcz_store.cc:276] 0810 18:52:40.447212 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:41.386789 (+939577us) service_pool.cc:227] Handling call
0810 18:52:41.408710 (+ 21921us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:41.781313 (+372603us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:41.781574 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:41.794152 21154 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:41.801883 16740 raft_consensus.cc:3037] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:41.772925 21678 raft_consensus.cc:482] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:41.806535 21568 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:41.804646 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 409661us
I20260810 18:52:41.392248 21654 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:41.692586 21461 leader_election.cc:288] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:41.811982 16573 leader_election.cc:302] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:41.693712 21668 log.cc:924] Time spent T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.344s user 0.001s sys 0.000s
I20260810 18:52:41.866782 16752 raft_consensus.cc:1261] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260810 18:52:41.590938 21613 raft_consensus.cc:482] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:41.720158 21493 raft_consensus.cc:482] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:41.827652 16502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1620) took 1360 ms. Trace:
W20260810 18:52:41.871799 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 96952.9us
I20260810 18:52:41.801442 21567 raft_consensus.cc:482] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:41.802493 16450 leader_election.cc:302] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
W20260810 18:52:41.879504 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 77112.1us
W20260810 18:52:41.879654 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 486368us
I20260810 18:52:41.834220 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:41.808105 21462 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.642s user 0.005s sys 0.016s
W20260810 18:52:41.842607 16701 leader_election.cc:334] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.665s (SENT)
I20260810 18:52:41.888885 16701 leader_election.cc:302] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:41.799443 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:41.792804 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:41.853075 16506 raft_consensus.cc:2454] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 8.
I20260810 18:52:41.889715 16747 raft_consensus.cc:2454] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
I20260810 18:52:41.808972 21170 raft_consensus.cc:2726] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260810 18:52:41.863183 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 47421.7us
I20260810 18:52:41.893344 16575 leader_election.cc:302] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:41.865219 16633 raft_consensus.cc:2454] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 9.
I20260810 18:52:41.798817 21518 raft_consensus.cc:686] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:41.413347 21553 raft_consensus.cc:504] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:41.867229 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 55344.7us
I20260810 18:52:41.867802 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:41.894626 21191 raft_consensus.cc:2726] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260810 18:52:41.903414 16628 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 10.
I20260810 18:52:41.824120 21444 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:41.825098 21498 raft_consensus.cc:2781] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:41.826777 21532 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:41.939334 21498 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:41.939790 21498 raft_consensus.cc:3037] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:41.939478 21532 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:41.941721 21532 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:41.943239 21532 raft_consensus.cc:482] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:41.826277 21667 raft_consensus.cc:2781] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:41.943133 21535 consensus_queue.cc:237] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:41.944223 21667 raft_consensus.cc:686] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:41.945092 21667 consensus_queue.cc:237] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:41.825279 21226 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:41.880918 16632 raft_consensus.cc:2396] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260810 18:52:41.873627 21524 raft_consensus.cc:2781] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader election won for term 7
I20260810 18:52:41.874435 16502 rpcz_store.cc:276] 0810 18:52:40.466731 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:40.466882 (+ 151us) service_pool.cc:227] Handling call
0810 18:52:40.475774 (+ 8892us) raft_consensus.cc:1457] Updating replica for 2 ops
0810 18:52:40.476151 (+ 377us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:40.476180 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0810 18:52:40.523717 (+ 47537us) log.cc:842] Serialized 52 byte log entry
0810 18:52:40.719797 (+196080us) raft_consensus.cc:1631] Marking committed up to 9
0810 18:52:40.738172 (+ 18375us) log.cc:842] Serialized 14 byte log entry
0810 18:52:40.778278 (+ 40106us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:40.778359 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:41.827355 (+1048996us) raft_consensus.cc:1671] finished
0810 18:52:41.827396 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:41.827573 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":229724,"prepare.queue_time_us":18128,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":40781,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":59223,"spinlock_wait_cycles":8386304,"thread_start_us":195735,"threads_started":1,"wal-append.queue_time_us":260243}
I20260810 18:52:41.888895 16450 leader_election.cc:302] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:52:41.956102 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 67286.4us
I20260810 18:52:41.889796 16501 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 8.
I20260810 18:52:41.890725 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:41.897204 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 54628.5us
I20260810 18:52:41.898741 16740 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 9.
I20260810 18:52:41.894292 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:41.901563 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:41.867208 21568 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:41.862912 21678 raft_consensus.cc:504] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:41.892838 21567 raft_consensus.cc:504] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:41.893101 21555 raft_consensus.cc:2781] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:41.911343 16622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 2136) took 1465 ms. Trace:
I20260810 18:52:41.906785 21539 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:41.879652 21613 raft_consensus.cc:504] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:41.846882 21512 raft_consensus.cc:504] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:41.892233 21493 raft_consensus.cc:504] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:41.923986 16507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1603) took 2284 ms. Trace:
I20260810 18:52:41.954849 21524 raft_consensus.cc:686] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:41.868821 21004 consensus_queue.cc:1046] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260810 18:52:41.985275 21004 raft_consensus.cc:3037] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:41.985090 21524 consensus_queue.cc:237] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:41.987541 21518 consensus_queue.cc:237] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:41.989898 21568 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:41.950136 16573 leader_election.cc:302] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:41.954005 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 126415us
I20260810 18:52:41.963194 16752 raft_consensus.cc:2454] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:41.963965 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:41.958098 16747 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 9.
I20260810 18:52:41.965081 16622 rpcz_store.cc:276] 0810 18:52:40.446354 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:41.340034 (+893680us) service_pool.cc:227] Handling call
0810 18:52:41.340400 (+ 366us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:41.340750 (+ 350us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:41.340785 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:41.341210 (+ 425us) log.cc:842] Serialized 26 byte log entry
0810 18:52:41.600502 (+259292us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:41.600559 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:41.600654 (+ 95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:41.911128 (+310474us) raft_consensus.cc:1671] finished
0810 18:52:41.911151 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:41.911285 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":45627,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":10173,"thread_start_us":258834,"threads_started":1,"wal-append.queue_time_us":561747}
I20260810 18:52:41.902241 21461 raft_consensus.cc:2726] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:52:41.898485 21665 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:41.931461 21391 raft_consensus.cc:2726] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:41.978974 16507 rpcz_store.cc:276] 0810 18:52:39.639382 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:39.639519 (+ 137us) service_pool.cc:227] Handling call
0810 18:52:39.646674 (+ 7155us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:39.720121 (+ 73447us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:39.720172 (+ 51us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:39.740498 (+ 20326us) log.cc:842] Serialized 26 byte log entry
0810 18:52:39.865422 (+124924us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:39.882603 (+ 17181us) log.cc:842] Serialized 14 byte log entry
0810 18:52:39.882850 (+ 247us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:39.882953 (+ 103us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:41.130664 (+1247711us) spinlock_profiling.cc:243] Waited 5.31 ms on lock 0x7b1800343a20. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1aee0718b1 00007f1aee07e661 00007f1ae51d2638 00007f1ae51d5b11 00007f1ae51d5ac9 00007f1ae51d5a59 00007f1ae51d5a21 00007f1ae51d4d1d 00007f1aee436204 00007f1aee436039 00007f1ae51b4dc6
0810 18:52:41.417080 (+286416us) spinlock_profiling.cc:243] Waited 4.63 ms on lock 0x7b44000232cc. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae50e4a95 00007f1ae50e54b2 00007f1ae7d8d511 00007f1ae7d88e1a 00007f1ae7d7cbee 00007f1ae7d7bf36 00007f1ae7d7bb58 00007f1ae7d85551 00007f1ae7d85509 00007f1ae7d85499
0810 18:52:41.923744 (+506664us) raft_consensus.cc:1671] finished
0810 18:52:41.923766 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:41.923908 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":16934,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":35797,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":5429,"spinlock_wait_cycles":22851712,"thread_start_us":165558,"threads_started":2,"wal-append.queue_time_us":191076}
W20260810 18:52:41.999208 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 98111.6us
I20260810 18:52:41.983896 21299 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:42.000162 16575 leader_election.cc:302] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:52:41.993780 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 43774.7us
I20260810 18:52:41.995610 16740 raft_consensus.cc:2396] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260810 18:52:41.992491 16628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 2140) took 1545 ms. Trace:
I20260810 18:52:42.001937 16628 rpcz_store.cc:276] 0810 18:52:40.447475 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:41.957075 (+1509600us) service_pool.cc:227] Handling call
0810 18:52:41.991849 (+ 34774us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:41.992062 (+ 213us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:41.992101 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:41.992154 (+ 53us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:41.992221 (+ 67us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:41.992288 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:41.992421 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:42.009173 16506 raft_consensus.cc:1226] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:42.003715 16573 leader_election.cc:302] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:41.977160 21555 raft_consensus.cc:482] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:42.026043 16628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 2146) took 1194 ms. Trace:
I20260810 18:52:41.943764 21532 raft_consensus.cc:504] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:42.034605 16508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1686) took 1165 ms. Trace:
I20260810 18:52:42.030310 16628 rpcz_store.cc:276] 0810 18:52:40.831716 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:42.024971 (+1193255us) service_pool.cc:227] Handling call
0810 18:52:42.025377 (+ 406us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:42.025614 (+ 237us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:42.025653 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:42.025710 (+ 57us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:42.025751 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:42.025816 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:42.025972 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:42.036896 16622 raft_consensus.cc:1261] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260810 18:52:41.998041 21226 raft_consensus.cc:2781] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:42.043041 21226 raft_consensus.cc:482] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:42.043526 21226 raft_consensus.cc:3037] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:42.044040 21539 consensus_queue.cc:1046] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260810 18:52:42.032091 16573 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:41.990576 21308 raft_consensus.cc:504] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:42.043458 16575 leader_election.cc:302] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:42.007246 21590 raft_consensus.cc:3032] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 6 LEADER]: Stepping down as leader of term 6
I20260810 18:52:42.041553 16622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 2145) took 1306 ms. Trace:
I20260810 18:52:42.049014 16622 rpcz_store.cc:276] 0810 18:52:40.735545 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:41.998795 (+1263250us) service_pool.cc:227] Handling call
0810 18:52:42.036492 (+ 37697us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:42.041266 (+ 4774us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:42.041496 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:42.025182 21495 leader_election.cc:288] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:42.033411 21461 raft_consensus.cc:2726] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260810 18:52:42.035887 21170 raft_consensus.cc:2781] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader election won for term 7
I20260810 18:52:42.033915 21555 raft_consensus.cc:3037] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:42.072582 16701 leader_election.cc:302] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:42.051911 16628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 2147) took 1209 ms. Trace:
I20260810 18:52:42.033610 21391 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:42.037361 21188 raft_consensus.cc:482] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
W20260810 18:52:42.077382 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 111570us
I20260810 18:52:42.048411 21640 leader_election.cc:288] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:41.995756 21570 consensus_queue.cc:1046] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20260810 18:52:42.076079 16628 rpcz_store.cc:276] 0810 18:52:40.842596 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:42.046551 (+1203955us) service_pool.cc:227] Handling call
0810 18:52:42.051405 (+ 4854us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:42.051584 (+ 179us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:42.051611 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:42.051651 (+ 40us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:42.051683 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:42.051735 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:42.051850 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:42.006458 21568 consensus_queue.cc:1035] T 2deb399ca7cb48198e6d3910bef67202 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260810 18:52:42.050607 21460 raft_consensus.cc:2781] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader election won for term 8
I20260810 18:52:42.051054 21590 raft_consensus.cc:729] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:42.008841 21665 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:42.046793 21380 leader_election.cc:288] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:42.101730 16621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60066 (request call id 321) took 2168 ms. Trace:
I20260810 18:52:42.074784 21685 raft_consensus.cc:2726] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:42.129523 16621 rpcz_store.cc:276] 0810 18:52:39.933332 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:39.933473 (+ 141us) service_pool.cc:227] Handling call
0810 18:52:39.933810 (+ 337us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:40.178473 (+244663us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:40.178519 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:40.178576 (+ 57us) raft_consensus.cc:1631] Marking committed up to 7
0810 18:52:40.178606 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:42.101419 (+1922813us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b54003ef3d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:42.101475 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:42.101633 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":514508800}
I20260810 18:52:42.130146 16450 leader_election.cc:302] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:42.131461 16621 raft_consensus.cc:1226] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 8. Current term is 9. Ops: []
I20260810 18:52:42.132215 16621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1723) took 1226 ms. Trace:
I20260810 18:52:42.132594 16621 rpcz_store.cc:276] 0810 18:52:40.905644 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:42.130829 (+1225185us) service_pool.cc:227] Handling call
0810 18:52:42.131235 (+ 406us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:42.131940 (+ 705us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:42.132153 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:42.104115 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:52:42.105587 21657 log.cc:924] Time spent T 50fbe8baa0b345b0b2c4e4489e397660 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.863s user 0.000s sys 0.001s
I20260810 18:52:42.106134 21552 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:42.108116 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:42.108681 21638 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:42.105612 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:42.110172 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:42.114250 16628 raft_consensus.cc:1226] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:42.126919 21437 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:42.124595 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 81230.3us
I20260810 18:52:42.158833 16575 leader_election.cc:302] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:42.159253 16628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1722) took 1253 ms. Trace:
I20260810 18:52:42.111440 16496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1614) took 1974 ms. Trace:
I20260810 18:52:42.148502 16624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 2131) took 2219 ms. Trace:
I20260810 18:52:42.084796 21170 raft_consensus.cc:686] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [term 7 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:42.154665 16450 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:42.152742 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:42.093103 21644 leader_election.cc:288] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:42.039661 16508 rpcz_store.cc:276] 0810 18:52:40.869447 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:41.795832 (+926385us) service_pool.cc:227] Handling call
0810 18:52:41.796113 (+ 281us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:41.796296 (+ 183us) raft_consensus.cc:1520] Early marking committed up to index 4
0810 18:52:41.868097 (+ 71801us) log.cc:842] Serialized 14 byte log entry
0810 18:52:42.034290 (+166193us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:42.034339 (+ 49us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:42.034375 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:42.034440 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:42.034550 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11391,"prepare.queue_time_us":107036,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":11564,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":109,"thread_start_us":222970,"threads_started":2,"wal-append.queue_time_us":86151}
I20260810 18:52:42.159624 16628 rpcz_store.cc:276] 0810 18:52:40.905341 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:42.113654 (+1208313us) service_pool.cc:227] Handling call
0810 18:52:42.114026 (+ 372us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:42.159003 (+ 44977us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:42.159196 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:42.170928 16450 leader_election.cc:302] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:42.156827 16621 raft_consensus.cc:1261] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260810 18:52:42.173894 16621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1724) took 1268 ms. Trace:
I20260810 18:52:42.129073 21640 raft_consensus.cc:2726] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:42.174312 16621 rpcz_store.cc:276] 0810 18:52:40.905917 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:42.133430 (+1227513us) service_pool.cc:227] Handling call
0810 18:52:42.133766 (+ 336us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:42.173608 (+ 39842us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:42.173831 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:42.175321 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:42.160063 16496 rpcz_store.cc:276] 0810 18:52:40.136982 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:40.147539 (+ 10557us) service_pool.cc:227] Handling call
0810 18:52:40.180698 (+ 33159us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:40.338404 (+157706us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:40.338439 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:40.338915 (+ 476us) log.cc:842] Serialized 26 byte log entry
0810 18:52:40.418841 (+ 79926us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:40.418886 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:42.111083 (+1692197us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b5400285d54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:42.111129 (+ 46us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:42.111172 (+ 43us) raft_consensus.cc:1671] finished
0810 18:52:42.111186 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:42.111384 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":346,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":82,"spinlock_wait_cycles":345578752,"thread_start_us":79549,"threads_started":1,"wal-append.queue_time_us":123832,"wal-append.run_cpu_time_us":11617,"wal-append.run_wall_time_us":1257097}
I20260810 18:52:42.160192 16624 rpcz_store.cc:276] 0810 18:52:39.928968 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:39.929892 (+ 924us) service_pool.cc:227] Handling call
0810 18:52:39.937860 (+ 7968us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:40.197625 (+259765us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:40.197655 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:40.198117 (+ 462us) log.cc:842] Serialized 26 byte log entry
0810 18:52:40.243596 (+ 45479us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:40.243646 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:41.151021 (+907375us) spinlock_profiling.cc:243] Waited 6.19 ms on lock 0x7b44000232cc. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae50e4a95 00007f1ae50e54b2 00007f1ae7d8d511 00007f1ae7d88e1a 00007f1ae7d7cbee 00007f1ae7d7bf36 00007f1ae7d7bb58 00007f1ae7d85551 00007f1ae7d85509 00007f1ae7d85499
0810 18:52:42.148093 (+997072us) spinlock_profiling.cc:243] Waited 247 ms on lock 0x7b54003683d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:42.148161 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:42.148224 (+ 63us) raft_consensus.cc:1671] finished
0810 18:52:42.148244 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:42.148423 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5830,"raft.queue_time_us":6029,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":6196,"spinlock_wait_cycles":581464064,"thread_start_us":27233,"threads_started":1,"wal-append.queue_time_us":43410}
I20260810 18:52:42.172096 16448 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:42.177094 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:42.080616 21391 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:42.179813 16450 leader_election.cc:302] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:42.182660 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:42.183792 16448 leader_election.cc:302] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:42.185674 16575 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:42.156224 21088 raft_consensus.cc:3037] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:42.093518 21188 raft_consensus.cc:504] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:42.188820 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:42.191555 16496 raft_consensus.cc:3037] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:42.182678 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:42.192111 16628 raft_consensus.cc:2454] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 8.
I20260810 18:52:42.193279 16624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1725) took 1287 ms. Trace:
I20260810 18:52:42.193776 16624 rpcz_store.cc:276] 0810 18:52:40.906194 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:42.176916 (+1270722us) service_pool.cc:227] Handling call
0810 18:52:42.192826 (+ 15910us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:42.192987 (+ 161us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:42.193010 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:42.193059 (+ 49us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:42.193095 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:42.193142 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:42.193229 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:42.158301 21460 raft_consensus.cc:686] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:42.193404 16628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 2150) took 1057 ms. Trace:
I20260810 18:52:42.128671 21359 consensus_queue.cc:1035] T 02553ba44ccf4839ae0c25e0b3a1b71b P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.052s
I20260810 18:52:42.197693 16628 rpcz_store.cc:276] 0810 18:52:41.136113 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:42.192752 (+1056639us) service_pool.cc:227] Handling call
0810 18:52:42.193015 (+ 263us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:42.193154 (+ 139us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:42.193173 (+ 19us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:42.193204 (+ 31us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:42.193231 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:42.193278 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:42.193359 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:42.155798 17780 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 8.
I20260810 18:52:42.198520 16573 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:42.199888 16628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 2152) took 1044 ms. Trace:
I20260810 18:52:42.206521 16628 rpcz_store.cc:276] 0810 18:52:41.155792 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:42.198958 (+1043166us) service_pool.cc:227] Handling call
0810 18:52:42.199355 (+ 397us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:42.199553 (+ 198us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:42.199581 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:42.199616 (+ 35us) raft_consensus.cc:1631] Marking committed up to 9
0810 18:52:42.199666 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:42.199714 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:42.199821 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:42.208811 16628 raft_consensus.cc:1226] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 8. Current term is 9. Ops: []
I20260810 18:52:42.172600 21654 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:42.216778 16575 leader_election.cc:302] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:42.219091 21308 leader_election.cc:288] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:42.220394 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
W20260810 18:52:42.220798 21717 log.cc:924] Time spent T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
W20260810 18:52:42.221019 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 148552us
I20260810 18:52:42.221974 16701 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:42.223671 21665 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:42.224028 16628 raft_consensus.cc:1261] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260810 18:52:42.155864 21592 consensus_queue.cc:1046] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 1, Time since last communication: 0.042s
I20260810 18:52:42.215134 21502 raft_consensus.cc:2726] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260810 18:52:42.225135 21678 leader_election.cc:288] T da1fa324d17c479383225a8cea494a80 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:42.186551 21711 raft_consensus.cc:2781] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:42.225703 21502 raft_consensus.cc:2726] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:42.225808 21711 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:42.226284 21711 raft_consensus.cc:3037] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:42.160416 21552 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:42.187306 21462 raft_consensus.cc:2781] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:42.227906 16623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1695) took 1818 ms. Trace:
I20260810 18:52:42.221053 16745 raft_consensus.cc:3037] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:42.160579 21638 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:42.172742 21537 consensus_queue.cc:1035] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.039s
I20260810 18:52:42.173116 21437 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:42.190471 21299 raft_consensus.cc:2781] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260810 18:52:42.228678 16495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52352 (request call id 1695) took 1105 ms. Trace:
I20260810 18:52:42.190624 21577 raft_consensus.cc:2781] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Leader election won for term 9
W20260810 18:52:42.341251 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 119359us
I20260810 18:52:42.354759 16701 leader_election.cc:302] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:42.305661 16496 raft_consensus.cc:2454] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:42.190080 21571 raft_consensus.cc:3037] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
W20260810 18:52:42.312824 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 96127.8us
W20260810 18:52:42.324165 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 125717us
I20260810 18:52:42.190646 21495 raft_consensus.cc:2726] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260810 18:52:42.206979 21685 raft_consensus.cc:2726] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260810 18:52:42.347851 16495 rpcz_store.cc:276] 0810 18:52:41.123092 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:41.232181 (+109089us) service_pool.cc:227] Handling call
0810 18:52:42.228513 (+996332us) spinlock_profiling.cc:243] Waited 532 ms on lock 0x7b5400137ed4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:42.228626 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1222560640}
W20260810 18:52:42.228192 16504 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52338 (request call id 1605) took 2587 ms (client timeout 1510 ms). Trace:
W20260810 18:52:42.436190 16504 rpcz_store.cc:269] 0810 18:52:39.640285 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:39.640626 (+ 341us) service_pool.cc:227] Handling call
0810 18:52:42.228045 (+2587419us) spinlock_profiling.cc:243] Waited 282 ms on lock 0x7b5400117954. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:42.228132 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":648019840}
W20260810 18:52:42.226387 16342 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55260 (request call id 124) took 18452 ms (client timeout 15000 ms). Trace:
I20260810 18:52:42.309237 16502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1624) took 1429 ms. Trace:
I20260810 18:52:42.312654 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:42.439615 16742 raft_consensus.cc:2454] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 7.
I20260810 18:52:42.347146 16623 rpcz_store.cc:276] 0810 18:52:40.409354 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:41.221894 (+812540us) service_pool.cc:227] Handling call
0810 18:52:42.227709 (+1005815us) spinlock_profiling.cc:243] Waited 186 ms on lock 0x7b540004ddd4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:42.227829 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":427182336}
I20260810 18:52:42.350930 16507 raft_consensus.cc:1261] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260810 18:52:42.442169 16507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1692) took 1542 ms. Trace:
I20260810 18:52:42.190536 21380 consensus_queue.cc:1046] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.122s
I20260810 18:52:42.365948 16573 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:42.443614 21678 raft_consensus.cc:3032] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 4 LEADER]: Stepping down as leader of term 4
I20260810 18:52:42.313498 21490 leader_election.cc:288] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:42.444010 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 78198.3us
I20260810 18:52:42.449949 16573 leader_election.cc:302] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:42.425879 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:42.437533 16342 rpcz_store.cc:269] 0810 18:52:23.774172 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:23.774404 (+ 232us) service_pool.cc:227] Handling call
0810 18:52:42.226291 (+18451887us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 18:52:39.204627 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 18:52:39.366565 (+161938us) write_op.cc:432] Acquiring schema lock in shared mode
0810 18:52:39.376329 (+ 9764us) write_op.cc:435] Acquired schema lock
0810 18:52:39.376351 (+ 22us) tablet.cc:660] Decoding operations
0810 18:52:39.400175 (+ 23824us) write_op.cc:620] Acquiring the partition lock for write op
0810 18:52:39.400268 (+ 93us) write_op.cc:641] Partition lock acquired for write op
0810 18:52:39.400294 (+ 26us) tablet.cc:683] Acquiring locks for 7 operations
0810 18:52:39.401131 (+ 837us) tablet.cc:699] Row locks acquired
0810 18:52:39.401165 (+ 34us) write_op.cc:260] PREPARE: finished
0810 18:52:39.401388 (+ 223us) write_op.cc:270] Start()
0810 18:52:39.401557 (+ 169us) write_op.cc:276] Timestamp: P: 1786387959401296 usec, L: 0
0810 18:52:39.401591 (+ 34us) op_driver.cc:331] REPLICATION: starting
0810 18:52:39.471015 (+ 69424us) log.cc:842] Serialized 2835 byte log entry
0810 18:52:39.802861 (+331846us) op_driver.cc:444] REPLICATION: finished
0810 18:52:39.937878 (+135017us) write_op.cc:301] APPLY: starting
0810 18:52:40.024092 (+ 86214us) tablet.cc:1365] starting BulkCheckPresence
0810 18:52:40.075915 (+ 51823us) tablet.cc:1368] finished BulkCheckPresence
0810 18:52:40.130826 (+ 54911us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 18:52:40.329286 (+198460us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 18:52:40.329644 (+ 358us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0810 18:52:40.329692 (+ 48us) write_op.cc:312] APPLY: finished
0810 18:52:40.330026 (+ 334us) log.cc:842] Serialized 59 byte log entry
0810 18:52:40.330219 (+ 193us) write_op.cc:489] Releasing partition, row and schema locks
0810 18:52:40.330335 (+ 116us) write_op.cc:454] Released schema lock
0810 18:52:40.330406 (+ 71us) write_op.cc:341] FINISH: Updating metrics
0810 18:52:40.330717 (+ 311us) write_op.cc:489] Releasing partition, row and schema locks
0810 18:52:40.330750 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":488590,"reported_tablets":20,"child_traces":[["op",{"apply.queue_time_us":104103,"apply.run_cpu_time_us":5015,"apply.run_wall_time_us":401591,"num_ops":7,"prepare.queue_time_us":339787,"prepare.run_cpu_time_us":16983,"prepare.run_wall_time_us":382880,"raft.queue_time_us":105405,"raft.run_cpu_time_us":17570,"raft.run_wall_time_us":70934,"replication_time_us":401169,"thread_start_us":287712,"threads_started":4,"wal-append.queue_time_us":139938,"wal-append.run_cpu_time_us":41950,"wal-append.run_wall_time_us":2101948}]]}
I20260810 18:52:42.443364 16622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 2148) took 1580 ms. Trace:
I20260810 18:52:42.401357 17786 raft_consensus.cc:2454] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:42.340297 21654 raft_consensus.cc:2726] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260810 18:52:42.444828 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 90164us
I20260810 18:52:42.434060 16575 leader_election.cc:302] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:42.437644 16502 rpcz_store.cc:276] 0810 18:52:40.880164 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:41.963567 (+1083403us) service_pool.cc:227] Handling call
0810 18:52:41.963892 (+ 325us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:41.964192 (+ 300us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:41.964230 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:41.976596 (+ 12366us) log.cc:842] Serialized 26 byte log entry
0810 18:52:42.088683 (+112087us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:42.088739 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:42.088832 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:42.220228 (+131396us) spinlock_profiling.cc:243] Waited 27.7 ms on lock 0x7b44000232cc. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae50e4a95 00007f1ae50e54b2 00007f1ae7d8d511 00007f1ae7d88e1a 00007f1ae7d7cbee 00007f1ae7d7bf36 00007f1ae7d7bb58 00007f1ae7d85551 00007f1ae7d85509 00007f1ae7d85499
0810 18:52:42.308891 (+ 88663us) raft_consensus.cc:1671] finished
0810 18:52:42.308912 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:42.309158 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":26635,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":95,"spinlock_wait_cycles":63640960,"thread_start_us":95236,"threads_started":1,"wal-append.queue_time_us":112309}
I20260810 18:52:42.347142 21462 raft_consensus.cc:482] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:42.229372 21512 leader_election.cc:288] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:42.276795 21493 leader_election.cc:288] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:42.360401 21667 raft_consensus.cc:2726] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260810 18:52:42.447892 17727 leader_election.cc:302] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:42.313148 21567 leader_election.cc:288] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:42.229035 21552 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:42.354499 21299 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:42.442655 16507 rpcz_store.cc:276] 0810 18:52:40.900057 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:41.998963 (+1098906us) service_pool.cc:227] Handling call
0810 18:52:42.343572 (+344609us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:42.441089 (+ 97517us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:42.441893 (+ 804us) spinlock_profiling.cc:243] Waited 231 ms on lock 0x7b5400137ed0. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:42.442090 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":530310400}
I20260810 18:52:42.451750 16701 leader_election.cc:302] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:42.450904 16622 rpcz_store.cc:276] 0810 18:52:40.862714 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:42.049960 (+1187246us) service_pool.cc:227] Handling call
0810 18:52:42.080974 (+ 31014us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:42.154708 (+ 73734us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:42.154739 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:42.154786 (+ 47us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:42.154811 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:42.443081 (+288270us) spinlock_profiling.cc:243] Waited 73.5 ms on lock 0x7b54003688d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:42.443141 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:42.443297 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":169083264}
I20260810 18:52:42.452353 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:42.451402 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:42.704114 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:42.557183 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:42.693841 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 259896us
I20260810 18:52:42.736166 16504 raft_consensus.cc:2379] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 6: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:42.558626 16621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 2149) took 1650 ms. Trace:
I20260810 18:52:42.425741 21665 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
W20260810 18:52:42.814584 16626 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1677) took 2467 ms (client timeout 2101 ms). Trace:
W20260810 18:52:42.825991 16626 rpcz_store.cc:269] 0810 18:52:40.268344 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:40.269666 (+ 1322us) service_pool.cc:227] Handling call
0810 18:52:42.694663 (+2424997us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540035f054. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:42.735931 (+ 41268us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2545908480}
I20260810 18:52:42.715839 16745 raft_consensus.cc:2454] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
I20260810 18:52:42.737198 16495 raft_consensus.cc:2454] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 4.
I20260810 18:52:42.829245 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:42.830965 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:42.842795 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 391080us
I20260810 18:52:42.844973 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:42.847647 21716 log.cc:924] Time spent T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.507s user 0.001s sys 0.000s
I20260810 18:52:42.458252 21380 raft_consensus.cc:2781] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260810 18:52:42.851384 21380 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:42.851861 21380 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:42.453774 21154 raft_consensus.cc:482] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:42.855525 21154 raft_consensus.cc:504] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:42.858266 21154 leader_election.cc:288] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:42.859791 21437 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:42.559409 21598 raft_consensus.cc:482] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:42.860881 21598 raft_consensus.cc:504] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:42.695304 21460 consensus_queue.cc:237] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:42.846993 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:42.817610 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:42.872650 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:42.872848 16507 raft_consensus.cc:2454] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 9.
I20260810 18:52:42.454748 21532 leader_election.cc:288] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:42.444079 21678 raft_consensus.cc:729] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:42.453362 21495 raft_consensus.cc:2781] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:42.457983 21462 raft_consensus.cc:3037] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:42.805982 16575 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:42.459446 21518 raft_consensus.cc:2781] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:42.902652 16496 raft_consensus.cc:1261] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260810 18:52:42.903297 16496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1628) took 1187 ms. Trace:
I20260810 18:52:42.903613 16496 rpcz_store.cc:276] 0810 18:52:41.715873 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:42.891949 (+1176076us) service_pool.cc:227] Handling call
0810 18:52:42.892209 (+ 260us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:42.903027 (+ 10818us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:42.903242 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:42.904887 16496 raft_consensus.cc:1261] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260810 18:52:42.905411 16496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1631) took 1159 ms. Trace:
I20260810 18:52:42.451786 21568 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:42.884788 16701 leader_election.cc:302] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:42.565383 21444 raft_consensus.cc:3032] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 8 LEADER]: Stepping down as leader of term 8
I20260810 18:52:42.808377 16621 rpcz_store.cc:276] 0810 18:52:40.908058 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:42.191709 (+1283651us) service_pool.cc:227] Handling call
0810 18:52:42.191989 (+ 280us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:42.192185 (+ 196us) raft_consensus.cc:1520] Early marking committed up to index 8
0810 18:52:42.216462 (+ 24277us) log.cc:842] Serialized 14 byte log entry
0810 18:52:42.557939 (+341477us) log.cc:842] Serialized 14 byte log entry
0810 18:52:42.558177 (+ 238us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:42.558243 (+ 66us) raft_consensus.cc:1631] Marking committed up to 8
0810 18:52:42.558305 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:42.558394 (+ 89us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:42.558548 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":106406,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":41,"raft.queue_time_us":83847,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":20103,"spinlock_wait_cycles":4941184,"thread_start_us":240700,"threads_started":1,"wal-append.queue_time_us":471263}
I20260810 18:52:42.889571 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:42.807744 21299 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 10 FOLLOWER]: Advancing to term 11
I20260810 18:52:42.458564 21577 raft_consensus.cc:686] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [term 9 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:42.937330 21577 consensus_queue.cc:237] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:42.939632 21598 leader_election.cc:288] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:42.457829 21640 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:42.900116 16507 raft_consensus.cc:1261] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
W20260810 18:52:42.940450 21498 consensus_meta.cc:287] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 1.000s user 0.010s sys 0.001s
I20260810 18:52:42.940878 16507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1693) took 1837 ms. Trace:
W20260810 18:52:42.940989 16702 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.702s user 0.014s sys 0.089s
W20260810 18:52:42.941439 16702 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.768s user 0.014s sys 0.089s
I20260810 18:52:42.941255 16507 rpcz_store.cc:276] 0810 18:52:41.103052 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:42.889763 (+1786711us) service_pool.cc:227] Handling call
0810 18:52:42.890081 (+ 318us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:42.940619 (+ 50538us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:42.940818 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:42.905696 16496 rpcz_store.cc:276] 0810 18:52:41.746853 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:42.904302 (+1157449us) service_pool.cc:227] Handling call
0810 18:52:42.904598 (+ 296us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:42.905205 (+ 607us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:42.905368 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:42.713508 21715 raft_consensus.cc:482] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:42.942847 21667 consensus_queue.cc:1035] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260810 18:52:42.943361 21715 raft_consensus.cc:504] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:42.944598 21667 raft_consensus.cc:2726] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 LEADER]: Leader election lost for term 4. Reason: could not achieve majority
I20260810 18:52:42.945438 21715 leader_election.cc:288] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:42.918514 21057 raft_consensus.cc:482] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:42.706890 21732 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:42.876209 21665 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:42.429486 21170 consensus_queue.cc:237] T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:42.964202 16448 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.772s user 0.048s sys 0.069s
W20260810 18:52:42.964617 16448 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.773s user 0.048s sys 0.069s
I20260810 18:52:42.921761 21678 consensus_queue.cc:260] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:42.822562 21552 raft_consensus.cc:2781] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260810 18:52:42.926106 21711 consensus_meta.cc:287] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.699s user 0.006s sys 0.003s
I20260810 18:52:42.935578 21524 raft_consensus.cc:482] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:42.883118 21695 log.cc:924] Time spent T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.704s user 0.000s sys 0.001s
I20260810 18:52:42.909173 21495 raft_consensus.cc:686] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:42.845042 21391 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:42.901353 21518 raft_consensus.cc:482] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:42.909243 21444 raft_consensus.cc:729] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:42.839177 21539 raft_consensus.cc:3037] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:42.712208 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:42.948443 16504 raft_consensus.cc:3037] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:42.908746 16695 leader_election.cc:302] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:42.890308 21696 log.cc:924] Time spent T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.780s user 0.000s sys 0.001s
W20260810 18:52:42.951179 17726 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.016s sys 0.021s
W20260810 18:52:42.964895 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 779975us
W20260810 18:52:42.966869 16447 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.006s sys 0.039s
I20260810 18:52:42.971709 21490 raft_consensus.cc:2726] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260810 18:52:42.971112 21552 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:42.884624 21537 raft_consensus.cc:482] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:42.950584 21673 log.cc:924] Time spent T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.077s user 0.000s sys 0.001s
I20260810 18:52:42.920567 21638 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:42.946192 21057 raft_consensus.cc:504] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:43.038005 16447 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.609s user 0.007s sys 0.039s
W20260810 18:52:43.038030 17726 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.593s user 0.017s sys 0.021s
W20260810 18:52:42.967636 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 123756us
W20260810 18:52:43.136590 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 699649us
I20260810 18:52:43.137073 16701 leader_election.cc:302] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
W20260810 18:52:43.044129 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 238243us
I20260810 18:52:43.001153 21518 raft_consensus.cc:3037] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
W20260810 18:52:43.046635 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 137946us
I20260810 18:52:43.142508 16695 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:43.111213 21493 raft_consensus.cc:482] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:43.113052 16502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:52330 (request call id 862) took 1884 ms. Trace:
W20260810 18:52:43.113615 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 74911.5us
I20260810 18:52:43.115284 21750 raft_consensus.cc:2781] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:52:42.950577 21732 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:43.045342 21739 raft_consensus.cc:482] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:43.116673 21462 raft_consensus.cc:504] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:43.117323 21710 log.cc:924] Time spent T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 1.021s user 0.000s sys 0.017s
I20260810 18:52:43.117622 21665 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:43.118052 16631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1691) took 2710 ms (client timeout 2216 ms). Trace:
I20260810 18:52:43.049841 16742 raft_consensus.cc:1261] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260810 18:52:43.124339 16624 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.009s sys 0.056s
I20260810 18:52:42.940881 21498 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:42.940478 21640 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:43.128695 16505 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:51572 (request call id 19) took 2300 ms. Trace:
I20260810 18:52:43.007313 21678 raft_consensus.cc:3037] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
W20260810 18:52:43.127269 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 679428us
I20260810 18:52:43.010818 16745 raft_consensus.cc:1261] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260810 18:52:43.138252 16575 leader_election.cc:302] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:42.972136 21568 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:43.051360 21552 raft_consensus.cc:3037] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:43.216035 16636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1685) took 2893 ms. Trace:
I20260810 18:52:43.243582 16636 rpcz_store.cc:276] 0810 18:52:40.322368 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:41.313663 (+991295us) service_pool.cc:227] Handling call
0810 18:52:41.348409 (+ 34746us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:41.786173 (+437764us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:43.215838 (+1429665us) spinlock_profiling.cc:243] Waited 15.3 ms on lock 0x7b540035dc50. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:43.215979 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":35215744}
I20260810 18:52:43.216612 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:43.245023 16749 raft_consensus.cc:3037] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Advancing to term 9
W20260810 18:52:43.239436 16624 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.667s user 0.010s sys 0.065s
I20260810 18:52:43.144917 16573 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:43.219656 17786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55586 (request call id 464) took 1111 ms. Trace:
I20260810 18:52:43.239540 16505 rpcz_store.cc:276] 0810 18:52:40.820466 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:40.837625 (+ 17159us) service_pool.cc:227] Handling call
0810 18:52:43.112802 (+2275177us) spinlock_profiling.cc:243] Waited 527 ms on lock 0x7b54001ca7d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f4bc3 00007f1aed03e81a 00007f1ae6731d04 00007f1ae6731c92 00007f1ae6731bc1 00007f1ae6731b3c 00007f1ae6730df2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:43.120431 (+ 7629us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1212832128}
I20260810 18:52:43.297245 17786 rpcz_store.cc:276] 0810 18:52:42.108190 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:42.109896 (+ 1706us) service_pool.cc:227] Handling call
0810 18:52:43.219446 (+1109550us) spinlock_profiling.cc:243] Waited 247 ms on lock 0x7b54005e0454. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:43.219584 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":567354752}
I20260810 18:52:42.977896 21524 raft_consensus.cc:504] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:43.181941 16504 raft_consensus.cc:2454] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 8.
I20260810 18:52:43.204932 16746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2152) took 1787 ms. Trace:
I20260810 18:52:43.296535 16624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 2151) took 2141 ms. Trace:
I20260810 18:52:43.248354 17727 leader_election.cc:302] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:43.121213 21715 raft_consensus.cc:482] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:43.308835 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 166413us
W20260810 18:52:43.308869 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 165340us
W20260810 18:52:43.313781 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 65552.3us
W20260810 18:52:43.239306 16631 rpcz_store.cc:269] 0810 18:52:40.407907 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:40.778283 (+370376us) service_pool.cc:227] Handling call
0810 18:52:43.117847 (+2339564us) spinlock_profiling.cc:243] Waited 969 ms on lock 0x7b54003845d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:43.117983 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2228802432}
I20260810 18:52:43.313504 16624 rpcz_store.cc:276] 0810 18:52:41.155186 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:42.195148 (+1039962us) service_pool.cc:227] Handling call
0810 18:52:42.215678 (+ 20530us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:42.215924 (+ 246us) raft_consensus.cc:1520] Early marking committed up to index 3
0810 18:52:42.327137 (+111213us) log.cc:842] Serialized 14 byte log entry
0810 18:52:43.296179 (+969042us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:43.296233 (+ 54us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:43.296276 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:43.296346 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:43.296475 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16178,"prepare.queue_time_us":111581,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":3701,"raft.queue_time_us":137838,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":16619,"thread_start_us":968576,"threads_started":2,"wal-append.queue_time_us":607359}
I20260810 18:52:43.316476 16749 raft_consensus.cc:2454] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 9.
W20260810 18:52:43.333755 16749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43738 (request call id 2159) took 1796 ms (client timeout 1770 ms). Trace:
I20260810 18:52:43.334005 16448 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:43.334111 16749 rpcz_store.cc:269] 0810 18:52:41.537796 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:41.788450 (+250654us) service_pool.cc:227] Handling call
0810 18:52:43.216450 (+1428000us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b5800270f98. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed03c1b0 00007f1aed03ba94 00007f1ae6723c04 00007f1ae6723b92
0810 18:52:43.333690 (+117240us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24452864}
I20260810 18:52:43.335314 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:43.316740 21498 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:43.316998 21692 log.cc:924] Time spent T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 1.340s user 0.000s sys 0.013s
I20260810 18:52:43.317406 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:43.220283 21743 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:43.318272 21732 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:43.352469 21732 consensus_queue.cc:1035] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:52:43.319340 21154 raft_consensus.cc:2781] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260810 18:52:43.319797 21462 leader_election.cc:288] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:43.319914 16632 raft_consensus.cc:1226] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 7. Current term is 8. Ops: [7.9-7.9]
I20260810 18:52:43.320149 21568 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:43.217625 21004 consensus_meta.cc:287] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 1.232s user 0.010s sys 0.000s
I20260810 18:52:43.319356 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:43.308590 21391 raft_consensus.cc:2726] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260810 18:52:43.322000 21590 consensus_queue.cc:260] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:43.323153 16495 raft_consensus.cc:2454] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
W20260810 18:52:43.323369 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 185220us
I20260810 18:52:43.318276 21640 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:43.213231 21730 log.cc:924] Time spent T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.652s user 0.001s sys 0.000s
I20260810 18:52:43.211704 21720 raft_consensus.cc:979] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea: : Instructing follower 980f343dd1fd4e049143f8b9d9f2f1be to start an election
W20260810 18:52:43.202764 21676 log.cc:924] Time spent T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 1.787s user 0.014s sys 0.008s
I20260810 18:52:43.215632 16496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:52330 (request call id 861) took 2056 ms. Trace:
I20260810 18:52:43.306772 16502 rpcz_store.cc:276] 0810 18:52:41.160020 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:42.899274 (+1739254us) service_pool.cc:227] Handling call
0810 18:52:43.044537 (+145263us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:43.338344 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:43.404497 17786 raft_consensus.cc:2396] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
W20260810 18:52:43.231124 21723 log.cc:924] Time spent T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 1.107s user 0.000s sys 0.010s
I20260810 18:52:43.308820 21685 raft_consensus.cc:2726] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:43.341668 16742 raft_consensus.cc:2454] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
I20260810 18:52:43.344947 21512 consensus_queue.cc:1035] T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
I20260810 18:52:43.279054 21750 raft_consensus.cc:482] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:42.979171 21711 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:43.134675 21490 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:43.347448 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 210475us
W20260810 18:52:43.309098 21088 consensus_meta.cc:287] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 1.111s user 0.011s sys 0.000s
I20260810 18:52:43.131121 21537 raft_consensus.cc:504] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:43.323192 21502 raft_consensus.cc:2726] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:52:43.351447 21743 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:43.313550 16746 rpcz_store.cc:276] 0810 18:52:41.417587 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:41.477025 (+ 59438us) service_pool.cc:227] Handling call
0810 18:52:43.203720 (+1726695us) spinlock_profiling.cc:243] Waited 25.7 ms on lock 0x7b5800270f98. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed0502b0 00007f1aed03b7f9 00007f1aed03b2a2 00007f1ae6721684
0810 18:52:43.204013 (+ 293us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:43.204328 (+ 315us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:43.204376 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:43.204430 (+ 54us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:43.204476 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:43.204549 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:43.204723 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":59161344}
I20260810 18:52:43.329295 17780 raft_consensus.cc:2354] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6.
I20260810 18:52:43.307559 21605 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:43.232282 21608 leader_election.cc:288] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:43.217737 21563 raft_consensus.cc:3032] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:52:43.358632 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:43.371381 16575 leader_election.cc:302] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:52:43.195053 21226 consensus_meta.cc:287] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 1.151s user 0.010s sys 0.001s
I20260810 18:52:43.376189 16695 leader_election.cc:302] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:43.446977 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 70913.4us
I20260810 18:52:43.370007 21004 raft_consensus.cc:2726] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:43.451089 17783 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55602 (request call id 425) took 2531 ms. Trace:
I20260810 18:52:43.214169 21495 consensus_queue.cc:237] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:43.140058 21444 consensus_queue.cc:260] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:43.426383 21750 raft_consensus.cc:3037] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:43.377995 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:43.378298 21731 raft_consensus.cc:2781] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260810 18:52:43.342855 21739 raft_consensus.cc:504] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:43.331930 21493 raft_consensus.cc:504] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:43.378953 21590 raft_consensus.cc:3037] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:43.344092 21715 raft_consensus.cc:504] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:43.435415 21490 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:43.469105 21715 leader_election.cc:288] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:43.469255 21519 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:43.379688 16750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2149) took 1963 ms. Trace:
I20260810 18:52:43.370144 21678 consensus_queue.cc:1035] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260810 18:52:43.355247 21513 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:43.469828 16750 rpcz_store.cc:276] 0810 18:52:41.416474 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:41.417741 (+ 1267us) service_pool.cc:227] Handling call
0810 18:52:41.418108 (+ 367us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:41.788954 (+370846us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:41.788995 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:41.789479 (+ 484us) log.cc:842] Serialized 26 byte log entry
0810 18:52:41.844699 (+ 55220us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:41.844743 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:41.844806 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:43.232373 (+1387567us) spinlock_profiling.cc:243] Waited 3.75 ms on lock 0x7b44000232cc. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae50e4a95 00007f1ae50e54b2 00007f1ae7d8d511 00007f1ae7d88e1a 00007f1ae7d7cbee 00007f1ae7d7bf36 00007f1ae7d7bb58 00007f1ae7d85551 00007f1ae7d85509 00007f1ae7d85499
0810 18:52:43.379309 (+146936us) raft_consensus.cc:1671] finished
0810 18:52:43.379344 (+ 35us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:43.379612 (+ 268us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":370422,"raft.queue_time_us":371807,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":111,"spinlock_wait_cycles":8627968,"thread_start_us":54781,"threads_started":1,"wal-append.queue_time_us":96727}
I20260810 18:52:43.471554 21490 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:43.401487 21555 consensus_meta.cc:287] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 1.308s user 0.013s sys 0.000s
I20260810 18:52:43.453583 21226 raft_consensus.cc:504] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:43.473166 21555 raft_consensus.cc:504] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:43.474481 21226 leader_election.cc:288] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:43.453507 17780 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55602 (request call id 433) took 1089 ms. Trace:
I20260810 18:52:43.475142 17780 rpcz_store.cc:276] 0810 18:52:42.364073 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:42.426297 (+ 62224us) service_pool.cc:227] Handling call
0810 18:52:43.453425 (+1027128us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:43.475672 21555 leader_election.cc:288] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:43.475984 21493 leader_election.cc:288] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:43.476639 16508 raft_consensus.cc:2379] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 6: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:43.458894 21731 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:43.480347 21731 raft_consensus.cc:3037] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:43.478184 21539 raft_consensus.cc:2726] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:43.483125 16745 raft_consensus.cc:2454] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
W20260810 18:52:43.425890 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 193698us
W20260810 18:52:43.434903 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 94751.7us
I20260810 18:52:43.436363 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:43.437302 16701 leader_election.cc:334] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.020s (SENT)
I20260810 18:52:43.376979 21299 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:43.488077 16701 leader_election.cc:302] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:43.488742 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:43.488945 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 51682.7us
I20260810 18:52:43.489324 21299 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 11 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:43.489526 16701 leader_election.cc:334] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.057s (SENT)
I20260810 18:52:43.489988 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:43.490396 16701 leader_election.cc:334] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.097s (SENT)
I20260810 18:52:43.490577 16742 raft_consensus.cc:2454] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
I20260810 18:52:43.421550 21628 raft_consensus.cc:1067] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Signalling peer 980f343dd1fd4e049143f8b9d9f2f1be to start an election
I20260810 18:52:43.492060 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:43.448051 16695 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:43.492718 16742 raft_consensus.cc:2454] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
W20260810 18:52:43.493340 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 45353us
I20260810 18:52:43.493582 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:43.494097 16742 raft_consensus.cc:3037] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:43.197060 21380 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:43.452024 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 80966us
I20260810 18:52:43.495178 16575 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:43.495828 21726 consensus_queue.cc:1035] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.151s
I20260810 18:52:43.496349 16575 leader_election.cc:302] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:43.496613 21380 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 9 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:43.497258 21380 raft_consensus.cc:2726] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260810 18:52:43.446159 21605 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:43.497988 16575 leader_election.cc:302] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:43.499429 21640 raft_consensus.cc:2726] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260810 18:52:43.451596 17783 rpcz_store.cc:276] 0810 18:52:40.909613 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:40.909730 (+ 117us) service_pool.cc:227] Handling call
0810 18:52:41.102339 (+192609us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:41.274898 (+172559us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:43.441005 (+2166107us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b54006a1654. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:43.441190 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":319236352}
I20260810 18:52:43.451475 21439 raft_consensus.cc:2781] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:43.498888 21743 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:43.459121 16501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1689) took 2583 ms. Trace:
I20260810 18:52:43.500344 21439 raft_consensus.cc:686] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 6 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:43.500809 21537 leader_election.cc:288] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:43.435230 21088 raft_consensus.cc:2726] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:43.499257 21605 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:43.501754 21226 raft_consensus.cc:2726] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260810 18:52:43.456171 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:43.440744 21188 leader_election.cc:288] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:43.446899 21563 raft_consensus.cc:729] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:43.357594 21154 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:43.471990 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:43.402498 16496 rpcz_store.cc:276] 0810 18:52:41.159695 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:42.942775 (+1783080us) service_pool.cc:227] Handling call
0810 18:52:43.214556 (+271781us) spinlock_profiling.cc:243] Waited 4.09 ms on lock 0x7b5800001898. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c726 00007f1aed08c900 00007f1aed03fb50 00007f1ae6736807 00007f1ae6736792 00007f1ae67366c1
0810 18:52:43.215537 (+ 981us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9410432}
I20260810 18:52:43.471118 21513 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:43.441708 16749 raft_consensus.cc:2454] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
I20260810 18:52:43.451826 21738 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:43.497291 21726 raft_consensus.cc:2726] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260810 18:52:43.455201 21733 raft_consensus.cc:2726] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260810 18:52:43.499588 21762 log.cc:924] Time spent T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.094s user 0.001s sys 0.000s
I20260810 18:52:43.440359 21644 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
W20260810 18:52:43.578732 21691 log.cc:924] Time spent T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 1.492s user 0.014s sys 0.000s
I20260810 18:52:43.615119 17786 raft_consensus.cc:1261] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260810 18:52:43.627547 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 128239us
I20260810 18:52:43.575374 21553 leader_election.cc:288] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:43.615013 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:43.551630 16502 raft_consensus.cc:1261] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 9. (term mismatch)
I20260810 18:52:43.616573 16750 raft_consensus.cc:2454] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 4.
I20260810 18:52:43.617039 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:43.633092 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:43.501318 21439 consensus_queue.cc:237] T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:43.527678 16752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 463) took 1388 ms. Trace:
I20260810 18:52:43.522764 21549 raft_consensus.cc:3037] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Advancing to term 7
W20260810 18:52:43.504402 16701 leader_election.cc:334] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.271s (SENT)
I20260810 18:52:43.639991 16701 leader_election.cc:302] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:43.526889 21613 leader_election.cc:288] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:43.536087 17781 raft_consensus.cc:1226] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 9 LEADER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 8. Current term is 9. Ops: []
I20260810 18:52:43.486860 21552 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:43.585346 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 236681us
I20260810 18:52:43.552964 21557 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:43.618868 21057 leader_election.cc:288] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:43.635684 21608 raft_consensus.cc:2781] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:43.620142 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:52:43.620944 16508 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52338 (request call id 1638) took 1516 ms (client timeout 1668 ms). Trace:
W20260810 18:52:43.663657 16508 rpcz_store.cc:269] 0810 18:52:42.105066 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:42.173280 (+ 68214us) service_pool.cc:227] Handling call
0810 18:52:43.620737 (+1447457us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b54000f6254. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:43.620878 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2973250816}
I20260810 18:52:43.621598 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:43.654429 16573 leader_election.cc:398] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Vote denied by peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387) with higher term. Message: Invalid argument: T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6.
I20260810 18:52:43.665203 16573 leader_election.cc:401] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260810 18:52:43.557505 21705 consensus_queue.cc:1035] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.066s
I20260810 18:52:43.513461 16632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 2142) took 3065 ms. Trace:
I20260810 18:52:43.512221 16633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 2143) took 3058 ms. Trace:
I20260810 18:52:43.670040 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:43.548357 21651 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:43.516701 16507 raft_consensus.cc:1261] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260810 18:52:43.459307 21359 consensus_queue.cc:1046] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 1, Time since last communication: 0.186s
I20260810 18:52:43.683254 16507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1702) took 1226 ms. Trace:
I20260810 18:52:43.683521 16626 raft_consensus.cc:2379] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 8: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:43.683795 16507 rpcz_store.cc:276] 0810 18:52:42.456572 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:42.942251 (+485679us) service_pool.cc:227] Handling call
0810 18:52:43.515998 (+573747us) spinlock_profiling.cc:243] Waited 4.52 ms on lock 0x7b5800001898. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed0502b0 00007f1aed03b7f9 00007f1aed03b2a2 00007f1ae6721684
0810 18:52:43.516254 (+ 256us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:43.682867 (+166613us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:43.683188 (+ 321us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10398976}
I20260810 18:52:43.474502 21739 leader_election.cc:288] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:43.501462 16575 leader_election.cc:302] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:43.540010 21524 leader_election.cc:288] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:43.685982 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 184608us
I20260810 18:52:43.526727 21461 raft_consensus.cc:482] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:43.691020 16743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2151) took 2273 ms. Trace:
I20260810 18:52:43.542723 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:43.691644 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:43.692749 17783 raft_consensus.cc:1226] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260810 18:52:43.568768 21699 raft_consensus.cc:3032] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Stepping down as leader of term 6
I20260810 18:52:43.697852 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:43.544476 16504 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52338 (request call id 1637) took 1440 ms. Trace:
I20260810 18:52:43.631412 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:43.500679 16501 rpcz_store.cc:276] 0810 18:52:40.876116 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:41.957197 (+1081081us) service_pool.cc:227] Handling call
0810 18:52:41.957605 (+ 408us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:41.962755 (+ 5150us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:41.962796 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:41.978868 (+ 16072us) log.cc:842] Serialized 26 byte log entry
0810 18:52:42.108401 (+129533us) raft_consensus.cc:1631] Marking committed up to 7
0810 18:52:42.108439 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:42.108510 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:43.230375 (+1121865us) spinlock_profiling.cc:243] Waited 2.14 ms on lock 0x7b44000232cc. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae50e4a95 00007f1ae50e54b2 00007f1ae7d8d511 00007f1ae7d88e1a 00007f1ae7d7cbee 00007f1ae7d7bf36 00007f1ae7d7bb58 00007f1ae7d85551 00007f1ae7d85509 00007f1ae7d85499
0810 18:52:43.453858 (+223483us) raft_consensus.cc:1671] finished
0810 18:52:43.453893 (+ 35us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:43.459038 (+ 5145us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":27844,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":118,"spinlock_wait_cycles":4920704,"thread_start_us":79485,"threads_started":1,"wal-append.queue_time_us":114121}
I20260810 18:52:43.632833 16502 pending_rounds.cc:76] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea: Aborting all ops after (but not including) 8
I20260810 18:52:43.700726 16502 pending_rounds.cc:97] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20260810 18:52:43.701011 16502 raft_consensus.cc:2866] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:52:43.658922 21726 raft_consensus.cc:2726] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260810 18:52:43.633911 17727 leader_election.cc:302] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:43.700043 17787 raft_consensus.cc:3037] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:43.657130 21685 raft_consensus.cc:482] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:43.707960 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 74134.8us
I20260810 18:52:43.707896 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:43.634882 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:43.636042 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:43.502301 21380 raft_consensus.cc:2726] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260810 18:52:43.460011 21502 raft_consensus.cc:2726] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260810 18:52:43.636898 16752 rpcz_store.cc:276] 0810 18:52:42.139089 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:42.143180 (+ 4091us) service_pool.cc:227] Handling call
0810 18:52:42.143537 (+ 357us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:42.182544 (+ 39007us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:42.182576 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:42.182615 (+ 39us) raft_consensus.cc:1631] Marking committed up to 7
0810 18:52:42.182646 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:43.527397 (+1344751us) spinlock_profiling.cc:243] Waited 38.8 ms on lock 0x7b540009ef54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:43.527459 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:43.527606 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":89245056}
I20260810 18:52:43.687913 21490 raft_consensus.cc:3037] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 6
I20260810 18:52:43.643244 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
W20260810 18:52:43.646394 16323 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 8810b6b0b676405daf5ffddc50a4d21f with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica d304ac47fdf846de81337057a4ef7bea is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
W20260810 18:52:43.540452 21760 log.cc:924] Time spent T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.085s user 0.002s sys 0.000s
W20260810 18:52:43.647624 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 143253us
I20260810 18:52:43.662784 17781 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55570 (request call id 555) took 2452 ms. Trace:
I20260810 18:52:43.641239 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:43.667390 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:43.670059 16632 rpcz_store.cc:276] 0810 18:52:40.448020 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:41.958195 (+1510175us) service_pool.cc:227] Handling call
0810 18:52:41.992728 (+ 34533us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:43.355607 (+1362879us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:43.513153 (+157546us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b54002a5654. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:43.513391 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3051692800}
I20260810 18:52:43.671962 16621 raft_consensus.cc:1226] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:43.670221 16633 rpcz_store.cc:276] 0810 18:52:40.453524 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:41.960609 (+1507085us) service_pool.cc:227] Handling call
0810 18:52:41.960977 (+ 368us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:42.028159 (+ 67182us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:42.028206 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:42.028263 (+ 57us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:42.028363 (+ 100us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:43.511935 (+1483572us) spinlock_profiling.cc:243] Waited 66.9 ms on lock 0x7b54003c7b54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:43.511996 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:43.512151 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":153928448}
I20260810 18:52:43.632766 21154 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Advancing to term 10
I20260810 18:52:43.503538 21711 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:43.687129 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:43.687209 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:43.687592 16575 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:43.572979 16495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52338 (request call id 1639) took 1466 ms. Trace:
I20260810 18:52:43.559753 21546 raft_consensus.cc:3032] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:43.695649 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:43.696595 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:43.691979 16745 raft_consensus.cc:2454] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 7.
I20260810 18:52:43.698670 16504 rpcz_store.cc:276] 0810 18:52:42.104354 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:42.824444 (+720090us) service_pool.cc:227] Handling call
0810 18:52:43.544268 (+719824us) spinlock_profiling.cc:243] Waited 37.8 ms on lock 0x7b54000e06d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:43.544403 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":86841728}
I20260810 18:52:43.743422 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:43.758774 16504 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af"
dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
mode: GRACEFUL
new_leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
from {username='slave'} at 127.0.0.1:51572
I20260810 18:52:43.581015 16748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 459) took 2354 ms. Trace:
I20260810 18:52:43.759615 16504 raft_consensus.cc:595] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Received request to transfer leadership to TS 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:43.633705 16746 raft_consensus.cc:2354] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6.
I20260810 18:52:43.658700 21733 raft_consensus.cc:482] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:43.700131 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:43.744428 17787 raft_consensus.cc:1261] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260810 18:52:43.754118 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:43.762945 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:43.752866 16750 raft_consensus.cc:3037] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:43.576406 16506 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1687) took 2701 ms. Trace:
I20260810 18:52:43.502241 21493 consensus_queue.cc:1035] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.185s
W20260810 18:52:43.756592 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 69105.4us
I20260810 18:52:43.704378 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:43.691432 16743 rpcz_store.cc:276] 0810 18:52:41.417214 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:41.428376 (+ 11162us) service_pool.cc:227] Handling call
0810 18:52:41.428751 (+ 375us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:41.786286 (+357535us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:41.786333 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:41.786899 (+ 566us) log.cc:842] Serialized 26 byte log entry
0810 18:52:41.845044 (+ 58145us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:42.081144 (+236100us) log.cc:842] Serialized 14 byte log entry
0810 18:52:42.123596 (+ 42452us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:42.123672 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:43.577951 (+1454279us) spinlock_profiling.cc:243] Waited 1.77 ms on lock 0x7b44000232cc. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae50e4a95 00007f1ae50e54b2 00007f1ae7d8d511 00007f1ae7d88e1a 00007f1ae7d7cbee 00007f1ae7d7bf36 00007f1ae7d7bb58 00007f1ae7d85551 00007f1ae7d85509 00007f1ae7d85499
0810 18:52:43.667317 (+ 89366us) raft_consensus.cc:1671] finished
0810 18:52:43.690802 (+ 23485us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:43.690959 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":399272,"prepare.queue_time_us":47808,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":452961,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":8153,"spinlock_wait_cycles":4069760,"thread_start_us":260214,"threads_started":2,"wal-append.queue_time_us":246409}
I20260810 18:52:43.764758 17778 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:43.764230 16506 rpcz_store.cc:276] 0810 18:52:40.875436 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:41.895415 (+1019979us) service_pool.cc:227] Handling call
0810 18:52:41.895703 (+ 288us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:42.024625 (+128922us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:43.576122 (+1551497us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b54001379d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:43.576335 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":260455168}
W20260810 18:52:43.511158 21724 log.cc:924] Time spent T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 1.402s user 0.005s sys 0.007s
I20260810 18:52:43.763896 16496 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:43.752290 17781 rpcz_store.cc:276] 0810 18:52:41.194107 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:43.535558 (+2341451us) service_pool.cc:227] Handling call
0810 18:52:43.535902 (+ 344us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:43.646457 (+110555us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:43.646703 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:43.756750 16495 rpcz_store.cc:276] 0810 18:52:42.106330 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:42.830642 (+724312us) service_pool.cc:227] Handling call
0810 18:52:43.572746 (+742104us) spinlock_profiling.cc:243] Waited 488 ms on lock 0x7b54001e14d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:43.572912 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1123280128}
I20260810 18:52:43.702433 16450 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:43.690634 21226 raft_consensus.cc:2726] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260810 18:52:43.636345 21644 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:43.728240 17785 raft_consensus.cc:3037] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Advancing to term 8
W20260810 18:52:43.728312 16323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 81966.3us
I20260810 18:52:43.729609 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:43.730535 21552 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:43.727285 16624 raft_consensus.cc:2454] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
I20260810 18:52:43.732739 21513 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:43.720247 21769 consensus_queue.cc:1035] T b650a523bfc649d4a7ec8bfd04a67c43 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260810 18:52:43.692408 21651 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:43.697826 21699 raft_consensus.cc:729] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:43.689338 21057 consensus_queue.cc:1035] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20260810 18:52:43.691211 21498 consensus_queue.cc:1035] T ab9362a1df854c4fad0204ee2f9f78ad P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.047s
I20260810 18:52:43.837813 16447 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:43.663030 21557 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:43.723825 21461 raft_consensus.cc:504] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:43.728302 21666 raft_consensus.cc:2781] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:43.757134 21546 raft_consensus.cc:729] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:43.760871 16504 raft_consensus.cc:603] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260810 18:52:43.699510 21605 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:43.737790 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:43.702167 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:43.859158 16502 raft_consensus.cc:3037] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:43.757450 16626 raft_consensus.cc:3037] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:43.726269 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:43.860404 16501 raft_consensus.cc:2454] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:43.762897 16636 raft_consensus.cc:2396] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260810 18:52:43.703627 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:43.765336 16575 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:43.754245 16742 raft_consensus.cc:2454] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 7.
I20260810 18:52:43.861660 16631 raft_consensus.cc:2454] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
W20260810 18:52:43.862210 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 96942.3us
I20260810 18:52:43.862854 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:43.798887 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:43.759265 16628 raft_consensus.cc:2454] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:43.864768 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:43.865352 16628 raft_consensus.cc:2454] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:43.866209 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:43.797231 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
W20260810 18:52:43.768075 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 65748.2us
I20260810 18:52:43.797709 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:43.770114 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:43.868515 16621 raft_consensus.cc:2454] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:43.813794 16633 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 10 FOLLOWER]: Advancing to term 11
I20260810 18:52:43.759583 16748 rpcz_store.cc:276] 0810 18:52:41.227057 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:41.227306 (+ 249us) service_pool.cc:227] Handling call
0810 18:52:41.244008 (+ 16702us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:41.806946 (+562938us) raft_consensus.cc:1520] Early marking committed up to index 9
0810 18:52:41.813624 (+ 6678us) log.cc:842] Serialized 14 byte log entry
0810 18:52:41.996287 (+182663us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:41.996363 (+ 76us) raft_consensus.cc:1631] Marking committed up to 9
0810 18:52:41.996420 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:43.580742 (+1584322us) spinlock_profiling.cc:243] Waited 563 ms on lock 0x7b54000708d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:43.580787 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:43.580933 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6212,"prepare.queue_time_us":16824,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":56,"raft.queue_time_us":565,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":26476,"spinlock_wait_cycles":1294087936,"thread_start_us":181682,"threads_started":1,"wal-append.queue_time_us":234132}
I20260810 18:52:43.818037 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:43.638240 21738 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:43.915558 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 47513.9us
I20260810 18:52:43.827416 16496 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 9.
I20260810 18:52:43.817256 16448 leader_election.cc:302] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:43.832695 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:43.836380 16749 raft_consensus.cc:2454] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 8.
I20260810 18:52:43.844584 17724 leader_election.cc:302] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:43.763087 21380 raft_consensus.cc:2726] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260810 18:52:43.947247 16746 raft_consensus.cc:2454] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
I20260810 18:52:43.948520 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:43.949259 16746 raft_consensus.cc:2454] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:43.767823 21726 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:43.952656 21726 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:43.953961 16633 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 11.
I20260810 18:52:43.954751 21726 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:43.809558 21444 raft_consensus.cc:3037] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:43.851248 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:43.725117 21524 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:43.968264 16500 raft_consensus.cc:2454] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:43.747709 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:43.857805 17786 raft_consensus.cc:2379] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 7: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
W20260810 18:52:43.752061 16701 leader_election.cc:334] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.447s (SENT)
I20260810 18:52:43.982344 16701 leader_election.cc:302] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:43.851747 21731 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:43.983517 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 231449us
I20260810 18:52:43.863173 16575 leader_election.cc:302] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:43.985499 21731 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:43.985734 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 122710us
I20260810 18:52:43.986508 16575 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:43.986621 21498 raft_consensus.cc:2781] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:43.876320 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:43.987308 21498 raft_consensus.cc:686] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:43.873293 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:43.988495 21498 consensus_queue.cc:237] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:43.989475 21739 raft_consensus.cc:2726] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:43.991361 21738 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:43.938201 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:43.991930 21493 raft_consensus.cc:2781] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader election won for term 7
I20260810 18:52:43.866801 16628 raft_consensus.cc:2454] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
I20260810 18:52:43.992444 16743 raft_consensus.cc:2454] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 8.
I20260810 18:52:43.809038 21502 consensus_queue.cc:1035] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.050s
I20260810 18:52:43.992900 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:43.993367 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:43.701838 21563 consensus_queue.cc:260] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:43.993947 16743 raft_consensus.cc:2454] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 7.
I20260810 18:52:43.994390 21563 raft_consensus.cc:3037] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:52:43.992473 21493 raft_consensus.cc:686] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:43.798521 21711 raft_consensus.cc:2781] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260810 18:52:43.831208 21518 consensus_meta.cc:287] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.550s user 0.010s sys 0.000s
I20260810 18:52:43.853629 21666 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:43.853838 21570 raft_consensus.cc:2726] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260810 18:52:43.959872 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:44.027410 16632 raft_consensus.cc:3037] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:43.864246 21038 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:43.958020 16450 leader_election.cc:302] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:43.870571 21170 raft_consensus.cc:2781] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Leader election won for term 9
W20260810 18:52:43.966761 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 149573us
I20260810 18:52:43.735021 21750 raft_consensus.cc:504] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:43.762079 21685 raft_consensus.cc:504] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:43.956480 21608 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:43.853385 21757 consensus_queue.cc:1035] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.119s
I20260810 18:52:43.927359 21460 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:43.973253 21748 raft_consensus.cc:2726] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260810 18:52:43.972839 16343 catalog_manager.cc:5495] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:52:43.973696 21524 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:43.974124 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:43.974594 21461 leader_election.cc:288] T 0c3053ea9475438bab46cba64f4cc2fa P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:43.967924 16750 raft_consensus.cc:2454] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
I20260810 18:52:43.977324 21557 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:43.871270 21605 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:43.979354 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:44.053624 16746 raft_consensus.cc:3037] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:43.987416 16742 raft_consensus.cc:2454] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:43.994905 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:44.000272 17727 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:43.972877 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:43.931232 16745 raft_consensus.cc:3037] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Advancing to term 11
I20260810 18:52:44.002705 16626 raft_consensus.cc:2396] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260810 18:52:43.854339 21613 consensus_queue.cc:1035] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.110s
I20260810 18:52:44.029928 16740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2157) took 2492 ms. Trace:
W20260810 18:52:44.060149 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 59952.4us
I20260810 18:52:44.059706 16740 rpcz_store.cc:276] 0810 18:52:41.537122 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:42.029028 (+491906us) service_pool.cc:227] Handling call
0810 18:52:42.043551 (+ 14523us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:42.043904 (+ 353us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:42.043931 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:42.044418 (+ 487us) log.cc:842] Serialized 26 byte log entry
0810 18:52:42.109666 (+ 65248us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:42.137098 (+ 27432us) log.cc:842] Serialized 14 byte log entry
0810 18:52:42.427740 (+290642us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:42.427849 (+ 109us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:43.510467 (+1082618us) spinlock_profiling.cc:243] Waited 14.3 ms on lock 0x7b44000232cc. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae50e4a95 00007f1ae50e54b2 00007f1ae7d8d511 00007f1ae7d88e1a 00007f1ae7d7cbee 00007f1ae7d7bf36 00007f1ae7d7bb58 00007f1ae7d85551 00007f1ae7d85509 00007f1ae7d85499
0810 18:52:43.999501 (+489034us) raft_consensus.cc:1671] finished
0810 18:52:44.029712 (+ 30211us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:44.029863 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33202,"prepare.queue_time_us":26944,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":56289,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":36787,"spinlock_wait_cycles":32893184,"thread_start_us":340104,"threads_started":2,"wal-append.queue_time_us":64465}
W20260810 18:52:44.034791 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 238549us
I20260810 18:52:44.035808 16631 raft_consensus.cc:1261] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260810 18:52:43.972886 21552 raft_consensus.cc:2726] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260810 18:52:43.798038 21733 raft_consensus.cc:504] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:43.978292 21699 consensus_queue.cc:260] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:44.051859 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:44.031934 21666 raft_consensus.cc:3037] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:44.063459 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:44.079483 16750 raft_consensus.cc:2454] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
I20260810 18:52:44.080996 16448 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:44.082180 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:44.066874 16573 leader_election.cc:302] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:44.059473 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 101537us
I20260810 18:52:44.064270 16752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43738 (request call id 2167) took 1067 ms. Trace:
I20260810 18:52:44.084606 16752 rpcz_store.cc:276] 0810 18:52:42.997218 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:43.955859 (+958641us) service_pool.cc:227] Handling call
0810 18:52:44.064205 (+108346us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:44.075086 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:44.038400 21608 raft_consensus.cc:3037] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:44.085260 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:44.086434 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:44.088291 16623 raft_consensus.cc:2379] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 8: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:44.088536 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:44.088860 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:44.089635 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:44.044812 21170 raft_consensus.cc:686] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:43.995997 21493 consensus_queue.cc:237] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:44.002749 21518 raft_consensus.cc:504] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:44.099491 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:44.362067 16628 raft_consensus.cc:2379] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 9: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:44.109113 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:44.362721 16633 raft_consensus.cc:2379] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 9: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:44.114850 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:44.032302 21038 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:44.090837 16343 catalog_manager.cc:5495] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 7 to 8, leader changed from d304ac47fdf846de81337057a4ef7bea (127.14.32.193) to 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194). New cstate: current_term: 8 leader_uuid: "0397d760589d40d3b92ef141cdfc2dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } }
I20260810 18:52:44.156385 16742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43738 (request call id 2171) took 1158 ms. Trace:
I20260810 18:52:44.365643 16742 rpcz_store.cc:276] 0810 18:52:42.998352 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:44.071275 (+1072923us) service_pool.cc:227] Handling call
0810 18:52:44.156271 (+ 84996us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:44.054554 21460 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:44.193969 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:44.217253 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 232929us
W20260810 18:52:44.369818 16701 connection.cc:232] RPC call timeout handler was delayed by -1.5892761768773198s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2315s for a response
W20260810 18:52:44.370213 16701 connection.cc:232] RPC call timeout handler was delayed by -1.3680893038399518s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2997s for a response
W20260810 18:52:44.371773 16701 connection.cc:232] RPC call timeout handler was delayed by -1.2083320969250053s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1718s for a response
I20260810 18:52:44.080087 21731 consensus_queue.cc:1046] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.103s
I20260810 18:52:44.343662 16745 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 11.
I20260810 18:52:44.344332 16752 raft_consensus.cc:1261] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20260810 18:52:44.345547 17726 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.938s (SENT) [suppressed 15 similar messages]
I20260810 18:52:44.391997 16745 raft_consensus.cc:1226] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 8. Current term is 9. Ops: [5.3-8.5]
I20260810 18:52:44.087047 16573 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:44.345683 16632 raft_consensus.cc:2396] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260810 18:52:44.393406 21170 consensus_queue.cc:237] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:44.074875 21557 raft_consensus.cc:2781] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:44.100484 21725 raft_consensus.cc:2781] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader election won for term 8
I20260810 18:52:44.393419 16631 raft_consensus.cc:1226] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 5. Current term is 6. Ops: []
W20260810 18:52:44.348439 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 264107us
I20260810 18:52:44.396237 16450 leader_election.cc:302] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:44.397018 21651 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:44.361310 21678 raft_consensus.cc:482] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:44.401213 21678 raft_consensus.cc:504] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:44.403402 21678 leader_election.cc:288] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:44.404383 21605 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:44.407954 16742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 468) took 1111 ms. Trace:
I20260810 18:52:44.408634 16745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2176) took 1075 ms. Trace:
I20260810 18:52:44.409026 16745 rpcz_store.cc:276] 0810 18:52:43.333038 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:44.391386 (+1058348us) service_pool.cc:227] Handling call
0810 18:52:44.391732 (+ 346us) raft_consensus.cc:1457] Updating replica for 3 ops
0810 18:52:44.408375 (+ 16643us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:44.408582 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:44.408339 16742 rpcz_store.cc:276] 0810 18:52:43.297013 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:44.366363 (+1069350us) service_pool.cc:227] Handling call
0810 18:52:44.407545 (+ 41182us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:44.407686 (+ 141us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:44.407709 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:44.407747 (+ 38us) raft_consensus.cc:1631] Marking committed up to 7
0810 18:52:44.407772 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:44.407818 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:44.407900 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:44.357384 21592 raft_consensus.cc:2781] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:44.350386 16746 raft_consensus.cc:2454] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 8.
I20260810 18:52:44.415998 16743 raft_consensus.cc:1226] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260810 18:52:44.416257 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:44.348850 16502 raft_consensus.cc:2454] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:44.098058 16748 raft_consensus.cc:2454] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
W20260810 18:52:44.351682 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 507158us
I20260810 18:52:44.030552 21546 consensus_queue.cc:260] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:44.352833 16575 connection.cc:232] RPC call timeout handler was delayed by -1.6878210189752281s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15910000000000002s for a response
W20260810 18:52:44.422297 16575 connection.cc:232] RPC call timeout handler was delayed by -1.686021019006148s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1593s for a response
W20260810 18:52:44.422636 16575 connection.cc:232] RPC call timeout handler was delayed by -1.4781210189685225s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1824s for a response
W20260810 18:52:44.423004 16575 connection.cc:232] RPC call timeout handler was delayed by -1.2369210189208388s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20920000000000002s for a response
W20260810 18:52:44.423333 16575 connection.cc:232] RPC call timeout handler was delayed by -1.1559210189152509s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2182s for a response
W20260810 18:52:44.423611 16575 connection.cc:232] RPC call timeout handler was delayed by -1.1415210189297795s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2198s for a response
W20260810 18:52:44.423923 16575 connection.cc:232] RPC call timeout handler was delayed by -1.1010210190434009s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2243s for a response
I20260810 18:52:44.424623 17727 leader_election.cc:302] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:44.425673 16343 catalog_manager.cc:5495] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 2 to 3, leader changed from 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 3 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:52:44.073570 21570 raft_consensus.cc:2781] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260810 18:52:44.355525 17785 consensus_meta.cc:287] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.545s user 0.011s sys 0.004s
I20260810 18:52:44.387785 16740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2175) took 1117 ms. Trace:
I20260810 18:52:44.427207 16740 rpcz_store.cc:276] 0810 18:52:43.270981 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:44.152548 (+881567us) service_pool.cc:227] Handling call
0810 18:52:44.387128 (+234580us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:44.387349 (+ 221us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:44.387386 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:44.387439 (+ 53us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:44.387481 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:44.387569 (+ 88us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:44.387716 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:44.388628 16628 raft_consensus.cc:1226] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 2. Current term is 3. Ops: []
I20260810 18:52:44.428905 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:44.428771 16742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 470) took 1087 ms. Trace:
I20260810 18:52:44.429682 16628 raft_consensus.cc:2379] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 5: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
W20260810 18:52:44.372123 16701 connection.cc:232] RPC call timeout handler was delayed by -1.1929262927733362s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31190000000000007s for a response
I20260810 18:52:44.430032 16742 rpcz_store.cc:276] 0810 18:52:43.340938 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:44.409947 (+1069009us) service_pool.cc:227] Handling call
0810 18:52:44.428357 (+ 18410us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:44.428521 (+ 164us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:44.428542 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:44.428575 (+ 33us) raft_consensus.cc:1631] Marking committed up to 8
0810 18:52:44.428601 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:44.428644 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:44.428722 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:44.418288 16752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2174) took 1279 ms. Trace:
W20260810 18:52:44.366941 21779 log.cc:924] Time spent T ef3f6c02785c4a76ba768b20f41f2dcd P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.284s user 0.007s sys 0.000s
I20260810 18:52:44.411962 16450 leader_election.cc:302] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:44.350279 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 269355us
W20260810 18:52:44.443594 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 356630us
I20260810 18:52:44.383569 21699 raft_consensus.cc:3037] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:44.434082 17785 raft_consensus.cc:2454] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 8.
I20260810 18:52:44.448774 17781 raft_consensus.cc:1226] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 7. Current term is 8. Ops: [7.9-7.9]
I20260810 18:52:44.433604 16752 rpcz_store.cc:276] 0810 18:52:43.139112 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:44.085225 (+946113us) service_pool.cc:227] Handling call
0810 18:52:44.343999 (+258774us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:44.417594 (+ 73595us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:44.418205 (+ 611us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:44.418515 16748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2177) took 1058 ms. Trace:
I20260810 18:52:44.449685 16748 rpcz_store.cc:276] 0810 18:52:43.359814 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:44.417746 (+1057932us) service_pool.cc:227] Handling call
0810 18:52:44.418043 (+ 297us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:44.418210 (+ 167us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:44.418231 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:44.418271 (+ 40us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:44.418300 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:44.418355 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:44.418465 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 18:52:44.433701 16701 leader_election.cc:334] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.844s (SENT)
I20260810 18:52:44.444414 16448 leader_election.cc:302] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:44.446055 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:44.451164 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:44.451726 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:44.450287 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:44.444382 16573 leader_election.cc:302] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:44.447244 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:44.453210 16745 raft_consensus.cc:2454] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 7.
I20260810 18:52:44.421846 21532 raft_consensus.cc:2781] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:44.455629 21532 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:44.455979 21532 raft_consensus.cc:3037] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:44.460008 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:44.461398 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:44.463012 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:44.389885 21738 raft_consensus.cc:482] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:44.364177 21549 consensus_meta.cc:287] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.713s user 0.011s sys 0.007s
I20260810 18:52:44.467031 16632 raft_consensus.cc:2396] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260810 18:52:44.411453 21557 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:44.470772 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:44.471446 16742 raft_consensus.cc:1261] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260810 18:52:44.468459 17727 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
W20260810 18:52:44.472359 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 53171us
I20260810 18:52:44.470935 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:44.472325 17771 raft_consensus.cc:1261] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260810 18:52:44.474541 16742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2178) took 1087 ms. Trace:
I20260810 18:52:44.475178 16631 raft_consensus.cc:2362] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6. Re-sending same reply.
I20260810 18:52:44.475210 16742 rpcz_store.cc:276] 0810 18:52:43.387257 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:44.446584 (+1059327us) service_pool.cc:227] Handling call
0810 18:52:44.471122 (+ 24538us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:44.474294 (+ 3172us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:44.474484 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:44.477349 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:44.477785 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:44.480288 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:44.481109 16633 raft_consensus.cc:2362] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5. Re-sending same reply.
I20260810 18:52:44.482211 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:44.486291 21460 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:44.487752 21704 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:44.487659 16343 catalog_manager.cc:5495] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb reported cstate change: term changed from 5 to 6, leader changed from 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196) to 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195). New cstate: current_term: 6 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:52:44.488349 21704 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:44.489799 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:44.490504 16506 raft_consensus.cc:2454] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 7.
I20260810 18:52:44.493322 21557 raft_consensus.cc:3037] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:44.495406 21685 leader_election.cc:288] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:44.496088 21549 raft_consensus.cc:2781] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:44.496587 21549 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:44.496963 21549 raft_consensus.cc:3037] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Advancing to term 8
W20260810 18:52:44.498008 21154 consensus_meta.cc:287] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.700s user 0.009s sys 0.017s
W20260810 18:52:44.499881 21777 log.cc:924] Time spent T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.138s user 0.000s sys 0.000s
I20260810 18:52:44.423686 21725 raft_consensus.cc:686] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [term 8 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:44.501528 16622 raft_consensus.cc:1226] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 8. Current term is 9. Ops: []
I20260810 18:52:44.415247 21592 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:44.503355 21592 raft_consensus.cc:3037] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:44.504139 21733 leader_election.cc:288] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:44.471885 21738 raft_consensus.cc:504] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
I20260810 18:52:44.506529 21738 leader_election.cc:288] T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:44.443463 21570 raft_consensus.cc:482] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:44.507391 21570 raft_consensus.cc:3037] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:44.507604 16746 raft_consensus.cc:1226] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:44.508315 16746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43726 (request call id 1611) took 1081 ms. Trace:
I20260810 18:52:44.508903 16632 raft_consensus.cc:1261] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260810 18:52:44.509958 16742 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7dd134517a994065a45633249a3ce405"
dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
from {username='slave'} at 127.0.0.1:43738
I20260810 18:52:44.510581 16742 raft_consensus.cc:482] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:52:44.510937 16742 raft_consensus.cc:3037] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:44.471809 21038 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:44.513576 16748 raft_consensus.cc:1261] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20260810 18:52:44.515460 16701 leader_election.cc:334] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.997s (SENT)
I20260810 18:52:44.515705 16573 leader_election.cc:302] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:44.515974 16701 leader_election.cc:302] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:44.517491 21733 raft_consensus.cc:2781] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:44.517942 21733 raft_consensus.cc:482] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:44.518347 21733 raft_consensus.cc:3037] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:44.502854 16622 raft_consensus.cc:1226] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:44.519699 16752 raft_consensus.cc:1226] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 4. Current term is 5. Ops: []
I20260810 18:52:44.520537 16752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2180) took 1077 ms. Trace:
I20260810 18:52:44.508646 16746 rpcz_store.cc:276] 0810 18:52:43.427281 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:44.447732 (+1020451us) service_pool.cc:227] Handling call
0810 18:52:44.507397 (+ 59665us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:44.508090 (+ 693us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:44.508266 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:44.520913 16752 rpcz_store.cc:276] 0810 18:52:43.443268 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:44.449895 (+1006627us) service_pool.cc:227] Handling call
0810 18:52:44.519493 (+ 69598us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:44.520267 (+ 774us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:44.520476 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:44.523943 16632 raft_consensus.cc:1261] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260810 18:52:44.524324 21705 consensus_queue.cc:1046] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260810 18:52:44.525502 21705 raft_consensus.cc:3032] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Stepping down as leader of term 7
W20260810 18:52:44.525658 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 113800us
I20260810 18:52:44.526005 21705 raft_consensus.cc:729] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:44.527169 16450 leader_election.cc:302] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:44.527339 21590 raft_consensus.cc:2781] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:44.527904 21590 raft_consensus.cc:482] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:44.528348 21590 raft_consensus.cc:3037] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:44.530483 21004 raft_consensus.cc:2781] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:44.531093 21004 raft_consensus.cc:686] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [term 6 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:44.532104 21004 consensus_queue.cc:237] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:44.411630 21644 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:44.543720 16628 raft_consensus.cc:2379] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 8: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:44.528256 16450 leader_election.cc:302] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:44.547868 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:44.549216 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 76023.3us
W20260810 18:52:44.550235 16701 leader_election.cc:334] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 3.119s (SENT)
I20260810 18:52:44.525492 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:44.502089 21725 consensus_queue.cc:237] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:44.530529 16573 leader_election.cc:302] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
W20260810 18:52:44.553735 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 108284us
I20260810 18:52:44.555598 16502 raft_consensus.cc:1261] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260810 18:52:44.557565 16633 raft_consensus.cc:2379] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 6: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:44.558013 16448 leader_election.cc:302] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:52:44.490839 17778 consensus_meta.cc:287] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.718s user 0.005s sys 0.002s
I20260810 18:52:44.560158 16450 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:44.560432 17778 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 9.
I20260810 18:52:44.561853 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:44.467363 21605 raft_consensus.cc:2781] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader election won for term 8
I20260810 18:52:44.473754 21296 raft_consensus.cc:3032] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Stepping down as leader of term 7
I20260810 18:52:44.564049 21188 raft_consensus.cc:2781] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:44.473555 21380 raft_consensus.cc:2781] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:44.567435 21524 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:44.448360 21598 raft_consensus.cc:482] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:44.568881 17770 raft_consensus.cc:1226] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 8. Current term is 9. Ops: []
I20260810 18:52:44.498483 21154 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:44.572132 21666 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:44.585415 16701 leader_election.cc:334] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 3.292s (SENT)
I20260810 18:52:44.574200 21557 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:44.576721 16450 leader_election.cc:302] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:44.566218 16742 raft_consensus.cc:504] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:44.620579 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 43978.4us
I20260810 18:52:44.621407 16450 leader_election.cc:302] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:44.578714 16448 leader_election.cc:302] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:44.577723 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:44.583549 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:44.584153 21188 raft_consensus.cc:482] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:44.578859 21296 raft_consensus.cc:729] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:44.579025 21748 raft_consensus.cc:2781] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:44.584053 21732 raft_consensus.cc:2781] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260810 18:52:44.666944 16742 leader_election.cc:288] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:44.584969 21380 raft_consensus.cc:482] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:44.602761 21666 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:44.573786 21592 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:44.681144 16747 raft_consensus.cc:1203] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 10 FOLLOWER]: Deduplicated request from leader. Original: 6.8->[8.9-8.9] Dedup: 8.9->[]
I20260810 18:52:44.681931 16747 raft_consensus.cc:1226] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 10 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 8. Current term is 10. Ops: [8.9-8.9]
I20260810 18:52:44.606046 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:44.564324 21685 raft_consensus.cc:2781] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:52:44.611480 21518 leader_election.cc:288] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:44.612898 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:44.614990 21563 consensus_meta.cc:287] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.620s user 0.010s sys 0.000s
W20260810 18:52:44.617616 21490 consensus_meta.cc:287] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.866s user 0.008s sys 0.004s
I20260810 18:52:44.583694 16573 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:44.699671 17785 raft_consensus.cc:1226] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 7. Current term is 8. Ops: []
I20260810 18:52:44.588264 21605 raft_consensus.cc:686] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 8 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:44.665055 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:44.666536 16507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:44.693354 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:44.694494 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:44.690516 16631 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 9.
W20260810 18:52:44.695520 16495 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.615s user 0.026s sys 0.067s
I20260810 18:52:44.697918 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:44.706043 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:44.711221 16495 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.630s user 0.027s sys 0.067s
I20260810 18:52:44.619577 16701 leader_election.cc:302] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:44.716112 21444 consensus_meta.cc:287] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.720s user 0.008s sys 0.015s
I20260810 18:52:44.716739 21444 raft_consensus.cc:2781] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:44.717087 21444 raft_consensus.cc:482] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:44.717411 21444 raft_consensus.cc:3037] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:44.717955 21590 raft_consensus.cc:504] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:44.663020 21188 raft_consensus.cc:3037] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:44.720175 16506 raft_consensus.cc:1226] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:44.667493 21769 raft_consensus.cc:2741] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260810 18:52:44.721323 16502 raft_consensus.cc:2379] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 6: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:44.710498 16508 raft_consensus.cc:2454] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 7.
I20260810 18:52:44.738294 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:44.680596 21359 raft_consensus.cc:2726] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260810 18:52:44.667913 21748 raft_consensus.cc:482] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:44.746338 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:44.545362 21705 consensus_queue.cc:260] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:44.753010 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 169422us
I20260810 18:52:44.759052 16573 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:44.680421 21732 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:44.680905 21380 raft_consensus.cc:3037] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:44.761816 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:52:44.768167 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 182852us
I20260810 18:52:44.773607 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:44.620164 21598 raft_consensus.cc:504] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:44.778841 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:44.780417 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:44.719694 21154 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 10 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:44.720347 21711 raft_consensus.cc:482] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:44.787150 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:44.794119 16496 raft_consensus.cc:2379] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 5: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:44.743168 21490 raft_consensus.cc:2726] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387) with higher term. Message: Invalid argument: T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6.
I20260810 18:52:44.719933 21685 raft_consensus.cc:686] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:44.751036 21748 raft_consensus.cc:3037] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:44.755990 21518 raft_consensus.cc:2726] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260810 18:52:44.781073 21608 consensus_meta.cc:287] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.621s user 0.011s sys 0.000s
I20260810 18:52:44.795641 16508 raft_consensus.cc:2379] T b4082c2e6a774acfa97ec268831748a5 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 5: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:44.794930 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:44.807132 16496 raft_consensus.cc:3032] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:44.807633 16496 raft_consensus.cc:729] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:44.805569 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:44.808772 21773 log.cc:924] Time spent T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.448s user 0.001s sys 0.002s
I20260810 18:52:44.809749 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:44.796214 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 450670us
I20260810 18:52:44.769245 21461 raft_consensus.cc:2781] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Leader election won for term 9
I20260810 18:52:44.769079 21732 raft_consensus.cc:3037] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:44.771675 21524 raft_consensus.cc:2781] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader election won for term 8
I20260810 18:52:44.802032 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:44.809767 16752 raft_consensus.cc:1261] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260810 18:52:44.850168 16752 pending_rounds.cc:76] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be: Aborting all ops after (but not including) 3
I20260810 18:52:44.824631 17726 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:44.795449 21711 raft_consensus.cc:3037] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:44.837642 16626 raft_consensus.cc:2454] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
I20260810 18:52:44.853135 17726 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:44.844686 16450 leader_election.cc:302] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:44.850427 16752 pending_rounds.cc:97] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260810 18:52:44.855384 16752 raft_consensus.cc:2866] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:52:44.783414 21705 raft_consensus.cc:3037] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:44.873983 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:44.874935 16742 raft_consensus.cc:2454] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
I20260810 18:52:44.857156 16450 leader_election.cc:302] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:44.847944 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:44.879637 16450 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:44.858973 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:44.849306 16496 consensus_queue.cc:260] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:44.881395 16746 raft_consensus.cc:3037] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:44.884428 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:44.885298 16742 raft_consensus.cc:2362] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 9. Re-sending same reply.
I20260810 18:52:44.886430 16695 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:44.765075 21296 consensus_queue.cc:260] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:44.854773 17726 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:44.861375 21801 raft_consensus.cc:482] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:44.920141 16695 leader_election.cc:302] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:44.915596 21557 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:44.895902 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:44.931437 16508 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Advancing to term 10
I20260810 18:52:44.892684 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:44.822172 21608 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:44.861531 21651 raft_consensus.cc:2726] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260810 18:52:44.943113 16624 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.008s sys 0.043s
I20260810 18:52:44.887493 21726 consensus_queue.cc:1035] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260810 18:52:44.941126 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:44.943500 16624 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.538s user 0.008s sys 0.043s
I20260810 18:52:44.883107 21738 raft_consensus.cc:2726] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:44.863526 21461 raft_consensus.cc:686] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 9 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:44.953071 16742 raft_consensus.cc:2454] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 9.
I20260810 18:52:44.962636 16746 raft_consensus.cc:2454] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
W20260810 18:52:44.887037 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 308377us
W20260810 18:52:44.914877 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 60208.4us
I20260810 18:52:44.872970 21524 raft_consensus.cc:686] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [term 8 LEADER]: Becoming Leader. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:44.953631 16624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 2180) took 1207 ms. Trace:
I20260810 18:52:44.919126 21563 raft_consensus.cc:2726] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260810 18:52:44.985562 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:44.986819 16746 raft_consensus.cc:2379] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 5: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:44.981719 16624 rpcz_store.cc:276] 0810 18:52:43.745819 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:43.931135 (+185316us) service_pool.cc:227] Handling call
0810 18:52:43.967164 (+ 36029us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:43.967421 (+ 257us) raft_consensus.cc:1520] Early marking committed up to index 3
0810 18:52:44.024785 (+ 57364us) log.cc:842] Serialized 14 byte log entry
0810 18:52:44.405368 (+380583us) spinlock_profiling.cc:243] Waited 5.11 ms on lock 0x7b1800299680. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1aee0718b1 00007f1aee07e661 00007f1ae51b4345 00007f1aedfeba39 00007f1ae51d18bb 00007f1ae51cfcbf 00007f1ae51d1e22 00007f1ae7d7b5bb 00007f1ae7d81063 00007f1ae7d8157b 00007f1ae810488a
0810 18:52:44.953272 (+547904us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:44.953326 (+ 54us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:44.953374 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:44.953455 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:44.953573 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":59008,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":5633,"raft.queue_time_us":3670,"spinlock_wait_cycles":11762560,"thread_start_us":547596,"threads_started":1}
W20260810 18:52:44.951112 21498 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.018s sys 0.031s
I20260810 18:52:44.823051 21605 consensus_queue.cc:237] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:44.975728 17726 leader_election.cc:302] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:44.989293 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:44.990010 16745 raft_consensus.cc:2379] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 7: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:44.938174 21590 leader_election.cc:288] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:44.927305 21808 raft_consensus.cc:2781] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:44.993777 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:45.008575 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:45.011695 16752 raft_consensus.cc:3037] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:45.011948 16742 raft_consensus.cc:2379] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 5: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:44.999260 16508 raft_consensus.cc:2454] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 10.
I20260810 18:52:44.879626 21532 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:44.955173 21296 raft_consensus.cc:3037] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:45.011581 16623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60066 (request call id 334) took 1014 ms. Trace:
I20260810 18:52:44.960107 21038 raft_consensus.cc:2781] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:45.013643 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:45.021169 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:45.021802 16742 raft_consensus.cc:2454] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:45.022107 16746 raft_consensus.cc:3037] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:45.022814 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:45.008790 21498 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.620s user 0.018s sys 0.031s
I20260810 18:52:44.960644 21739 raft_consensus.cc:2726] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260810 18:52:45.010308 21666 raft_consensus.cc:2726] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260810 18:52:44.985234 21592 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:45.024606 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:45.024844 17787 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s user 0.027s sys 0.017s
I20260810 18:52:45.023429 16742 raft_consensus.cc:2454] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
W20260810 18:52:45.029265 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 466951us
I20260810 18:52:45.036551 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:45.012938 21502 raft_consensus.cc:2781] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260810 18:52:45.037707 17787 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.578s user 0.028s sys 0.017s
I20260810 18:52:45.011365 21808 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:45.038791 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:45.022715 21570 consensus_meta.cc:287] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.512s user 0.007s sys 0.005s
I20260810 18:52:45.013542 21154 raft_consensus.cc:465] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 LEADER]: Not starting pre-election -- already a leader
I20260810 18:52:45.018544 21590 raft_consensus.cc:2781] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 11 FOLLOWER]: Leader election won for term 11
W20260810 18:52:45.045203 21804 log.cc:924] Time spent T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260810 18:52:44.938655 21801 raft_consensus.cc:504] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:45.046057 16740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 471) took 1680 ms. Trace:
I20260810 18:52:45.060415 16740 rpcz_store.cc:276] 0810 18:52:43.365335 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:44.428153 (+1062818us) service_pool.cc:227] Handling call
0810 18:52:44.471970 (+ 43817us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:44.823537 (+351567us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:44.823584 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:44.823627 (+ 43us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:44.823653 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:45.045785 (+222132us) spinlock_profiling.cc:243] Waited 351 ms on lock 0x7b54003bea54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:45.045844 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:45.045987 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":807920512}
I20260810 18:52:44.926996 21685 consensus_queue.cc:237] T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:45.046159 21592 raft_consensus.cc:2781] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election won for term 7
I20260810 18:52:45.073683 16746 raft_consensus.cc:2454] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 8.
W20260810 18:52:45.012320 21439 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.607s user 0.018s sys 0.037s
I20260810 18:52:45.050501 21808 raft_consensus.cc:3037] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:45.058905 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:45.060662 21188 raft_consensus.cc:504] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:45.056461 21731 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.006s sys 0.021s
W20260810 18:52:45.071704 17780 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.503s user 0.013s sys 0.032s
I20260810 18:52:45.054828 21598 leader_election.cc:288] T 91b6af1458cd44d8ba26b43579613ecb P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:45.048758 21666 consensus_queue.cc:1046] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
I20260810 18:52:45.046319 21490 consensus_queue.cc:1035] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.028s
I20260810 18:52:45.089018 16747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2156) took 3552 ms. Trace:
I20260810 18:52:45.089388 16747 rpcz_store.cc:276] 0810 18:52:41.536784 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:41.998269 (+461485us) service_pool.cc:227] Handling call
0810 18:52:43.228944 (+1230675us) spinlock_profiling.cc:243] Waited 1.28 ms on lock 0x7b5800270f98. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed0502b0 00007f1aed03b7f9 00007f1aed03b2a2 00007f1ae6721684
0810 18:52:43.581155 (+352211us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:44.682345 (+1101190us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:45.088092 (+405747us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54000708d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:45.088821 (+ 729us) spinlock_profiling.cc:243] Waited 352 ms on lock 0x7b54000708d0. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:45.088964 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3341786880}
I20260810 18:52:45.090101 21795 raft_consensus.cc:2726] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260810 18:52:45.084595 21731 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.549s user 0.006s sys 0.022s
I20260810 18:52:45.091542 17781 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55570 (request call id 565) took 1248 ms. Trace:
I20260810 18:52:45.091991 17781 rpcz_store.cc:276] 0810 18:52:43.843258 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:43.843436 (+ 178us) service_pool.cc:227] Handling call
0810 18:52:44.448547 (+605111us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:44.449190 (+ 643us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:45.091315 (+642125us) spinlock_profiling.cc:243] Waited 584 ms on lock 0x7b540062ff50. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:45.091473 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1342732032}
I20260810 18:52:45.096520 21704 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:45.097524 16749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2170) took 2099 ms. Trace:
I20260810 18:52:45.098184 21057 consensus_queue.cc:1035] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260810 18:52:45.098385 21835 consensus_queue.cc:1035] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260810 18:52:45.097905 16749 rpcz_store.cc:276] 0810 18:52:42.998060 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:44.074001 (+1075941us) service_pool.cc:227] Handling call
0810 18:52:44.074353 (+ 352us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:44.343775 (+269422us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:44.343807 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:44.344269 (+ 462us) log.cc:842] Serialized 26 byte log entry
0810 18:52:44.878107 (+533838us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:44.878160 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:44.879734 (+ 1574us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:45.097299 (+217565us) raft_consensus.cc:1671] finished
0810 18:52:45.097319 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:45.097469 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":17012,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":11051,"thread_start_us":467384,"threads_started":1,"wal-append.queue_time_us":731748}
I20260810 18:52:45.102469 16626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60066 (request call id 336) took 1104 ms. Trace:
I20260810 18:52:45.104844 21502 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting leader election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:45.105456 21502 raft_consensus.cc:3037] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:45.061687 21801 leader_election.cc:288] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:45.108644 17771 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55570 (request call id 551) took 4422 ms. Trace:
I20260810 18:52:45.109022 17771 rpcz_store.cc:276] 0810 18:52:40.686366 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:40.686521 (+ 155us) service_pool.cc:227] Handling call
0810 18:52:40.688473 (+ 1952us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:44.475119 (+3786646us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:45.108380 (+633261us) spinlock_profiling.cc:243] Waited 3.78 s on lock 0x7b540069da54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:45.108550 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8701314432}
I20260810 18:52:45.111261 21739 consensus_queue.cc:1046] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 18:52:45.112389 21739 consensus_queue.cc:1035] T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260810 18:52:45.113811 21810 raft_consensus.cc:3032] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 LEADER]: Stepping down as leader of term 8
I20260810 18:52:45.062152 21461 consensus_queue.cc:237] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:45.114224 21810 raft_consensus.cc:729] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:45.115114 21810 consensus_queue.cc:260] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:45.116472 21810 raft_consensus.cc:3037] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:45.119678 16504 raft_consensus.cc:1226] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:45.120613 21188 leader_election.cc:288] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:45.112442 21518 consensus_queue.cc:1046] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 18:52:45.122283 21518 raft_consensus.cc:3032] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Stepping down as leader of term 6
I20260810 18:52:45.071481 21498 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:45.064373 21590 raft_consensus.cc:686] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [term 11 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:45.102939 16626 rpcz_store.cc:276] 0810 18:52:43.997961 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:44.074772 (+ 76811us) service_pool.cc:227] Handling call
0810 18:52:45.102276 (+1027504us) spinlock_profiling.cc:243] Waited 719 ms on lock 0x7b54000ce254. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:45.102406 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1652702464}
I20260810 18:52:45.053308 21570 raft_consensus.cc:504] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:45.055836 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:45.051529 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:45.140003 16742 raft_consensus.cc:2379] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 5: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:45.141151 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:45.147698 16495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1645) took 1479 ms. Trace:
I20260810 18:52:45.169929 16495 rpcz_store.cc:276] 0810 18:52:43.668765 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:43.766933 (+ 98168us) service_pool.cc:227] Handling call
0810 18:52:43.829745 (+ 62812us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:43.996947 (+167202us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:43.996981 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:44.006375 (+ 9394us) log.cc:842] Serialized 26 byte log entry
0810 18:52:44.711661 (+705286us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:44.711701 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:45.108432 (+396731us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b54001e05d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:45.108472 (+ 40us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:45.131780 (+ 23308us) raft_consensus.cc:1671] finished
0810 18:52:45.131808 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:45.147610 (+ 15802us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25206,"raft.queue_time_us":96,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":7989,"spinlock_wait_cycles":383712128,"thread_start_us":630812,"threads_started":1,"wal-append.queue_time_us":877866}
I20260810 18:52:45.143314 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:45.152667 16343 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.014s sys 0.031s
I20260810 18:52:45.165031 16701 leader_election.cc:302] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:45.174494 16343 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.574s user 0.014s sys 0.031s
I20260810 18:52:45.175242 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:45.177239 16742 raft_consensus.cc:1261] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260810 18:52:45.178793 16742 raft_consensus.cc:1226] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260810 18:52:45.122684 21518 raft_consensus.cc:729] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:45.180228 16701 leader_election.cc:302] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:45.181485 16701 leader_election.cc:302] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:45.078189 21439 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.673s user 0.018s sys 0.037s
W20260810 18:52:45.137873 21493 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.022s sys 0.022s
I20260810 18:52:45.168900 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:45.166666 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:45.062069 21524 consensus_queue.cc:237] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:45.191033 21726 consensus_queue.cc:1046] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260810 18:52:45.194414 16747 raft_consensus.cc:2454] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 2.
I20260810 18:52:45.197085 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:45.192193 16701 leader_election.cc:302] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:45.201975 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:45.215812 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:45.172349 21608 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:45.184795 21598 raft_consensus.cc:3037] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:45.170285 21532 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:45.128571 21498 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:45.221462 16508 raft_consensus.cc:2379] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 6: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
W20260810 18:52:45.197135 21493 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.587s user 0.022s sys 0.022s
W20260810 18:52:45.224692 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 342804us
I20260810 18:52:45.193547 21806 raft_consensus.cc:2726] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260810 18:52:45.207590 17771 raft_consensus.cc:2379] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 7: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:52:45.196980 21154 raft_consensus.cc:2726] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:45.245560 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:45.267885 16450 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:45.218324 21739 raft_consensus.cc:2726] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260810 18:52:45.276078 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:45.277369 16748 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.004s sys 0.012s
W20260810 18:52:45.277792 16748 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.510s user 0.004s sys 0.012s
I20260810 18:52:45.283979 16695 leader_election.cc:302] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
W20260810 18:52:45.299500 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 78856.8us
I20260810 18:52:45.299790 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:45.305334 16508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
W20260810 18:52:45.308357 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 306665us
W20260810 18:52:45.311218 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 347193us
I20260810 18:52:45.317504 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:45.332536 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:45.283948 21726 consensus_queue.cc:1225] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer 0397d760589d40d3b92ef141cdfc2dbb log is divergent from this leader: its last log entry 7.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260810 18:52:45.320007 21592 raft_consensus.cc:686] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:45.335438 21839 consensus_queue.cc:1046] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260810 18:52:45.343370 21726 consensus_queue.cc:1035] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.008s
I20260810 18:52:45.349385 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:45.352694 16747 raft_consensus.cc:3037] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Advancing to term 8
W20260810 18:52:45.355659 21820 log.cc:924] Time spent T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.183s user 0.001s sys 0.000s
I20260810 18:52:45.342403 21592 consensus_queue.cc:237] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:45.359014 21819 log.cc:924] Time spent T 3e58f8d2231b4e2d900ec099ae700d09 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.133s user 0.000s sys 0.000s
I20260810 18:52:45.358937 17783 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55602 (request call id 438) took 1293 ms. Trace:
I20260810 18:52:45.359856 17783 rpcz_store.cc:276] 0810 18:52:44.065189 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:44.065306 (+ 117us) service_pool.cc:227] Handling call
0810 18:52:44.065615 (+ 309us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:44.348548 (+282933us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:44.348581 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:44.358033 (+ 9452us) log.cc:842] Serialized 26 byte log entry
0810 18:52:44.811790 (+453757us) raft_consensus.cc:1631] Marking committed up to 10
0810 18:52:45.221909 (+410119us) log.cc:842] Serialized 14 byte log entry
0810 18:52:45.222185 (+ 276us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:45.222258 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:45.358623 (+136365us) raft_consensus.cc:1671] finished
0810 18:52:45.358655 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:45.358853 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":491093,"raft.queue_time_us":328030,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":22010,"thread_start_us":996771,"threads_started":3,"wal-append.queue_time_us":717646}
I20260810 18:52:45.341527 16752 raft_consensus.cc:2454] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 8.
I20260810 18:52:45.366708 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:45.367666 16752 raft_consensus.cc:3037] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:45.337144 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:45.227988 21590 consensus_queue.cc:237] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:45.412134 16495 raft_consensus.cc:1226] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:45.320258 21806 raft_consensus.cc:2781] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 10 FOLLOWER]: Leader election won for term 10
I20260810 18:52:45.284791 21188 consensus_queue.cc:1035] T 156c7477728b4962a7069618cdf12b92 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.080s
I20260810 18:52:45.434427 16743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2172) took 2435 ms. Trace:
I20260810 18:52:45.434929 16743 rpcz_store.cc:276] 0810 18:52:42.998636 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:44.075578 (+1076942us) service_pool.cc:227] Handling call
0810 18:52:44.415759 (+340181us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:44.416413 (+ 654us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:45.434147 (+1017734us) spinlock_profiling.cc:243] Waited 340 ms on lock 0x7b54002ee5d0. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:45.434351 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":781357440}
I20260810 18:52:45.439128 16505 raft_consensus.cc:1226] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 7. Current term is 8. Ops: []
I20260810 18:52:45.442169 16752 raft_consensus.cc:2454] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
I20260810 18:52:45.408411 21004 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:45.315773 21154 consensus_queue.cc:1035] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.109s
I20260810 18:52:45.389146 21848 raft_consensus.cc:2781] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:45.459916 21848 raft_consensus.cc:482] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:45.460366 21848 raft_consensus.cc:3037] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:45.451577 16695 leader_election.cc:302] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:45.432983 21549 consensus_meta.cc:287] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.936s user 0.005s sys 0.008s
W20260810 18:52:45.458762 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 137654us
I20260810 18:52:45.464495 21549 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:45.464819 16701 leader_election.cc:302] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:45.467358 21549 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:45.370118 21739 consensus_queue.cc:1046] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.074s
W20260810 18:52:45.468741 16323 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 8810b6b0b676405daf5ffddc50a4d21f with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica d304ac47fdf846de81337057a4ef7bea is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 9 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:52:45.454615 21359 raft_consensus.cc:2726] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:45.433974 21570 leader_election.cc:288] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:45.442385 21750 leader_election.cc:288] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:45.310007 21518 consensus_queue.cc:260] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:45.483501 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:45.457731 21461 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:45.454563 21828 consensus_queue.cc:1035] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.089s
I20260810 18:52:45.561578 16505 raft_consensus.cc:1261] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260810 18:52:45.471951 21493 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:45.544143 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:45.573513 16450 leader_election.cc:302] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:45.575752 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:45.591813 16742 raft_consensus.cc:1226] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:45.484840 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:45.576448 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 111675us
W20260810 18:52:45.588047 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 272899us
I20260810 18:52:45.484256 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:45.603806 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 152311us
W20260810 18:52:45.580040 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 272961us
I20260810 18:52:45.601704 16624 raft_consensus.cc:2379] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 7: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:45.606590 16505 pending_rounds.cc:76] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea: Aborting all ops after (but not including) 4
I20260810 18:52:45.606803 16695 leader_election.cc:302] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:45.607124 16505 pending_rounds.cc:97] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260810 18:52:45.607673 16505 raft_consensus.cc:2866] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260810 18:52:45.562841 21733 consensus_meta.cc:287] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 1.044s user 0.005s sys 0.010s
I20260810 18:52:45.560356 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:45.609499 16752 raft_consensus.cc:3037] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:45.605626 16701 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:45.619488 16636 raft_consensus.cc:2396] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
W20260810 18:52:45.561810 21380 consensus_meta.cc:287] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.778s user 0.010s sys 0.000s
I20260810 18:52:45.636948 16748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 481) took 1027 ms. Trace:
I20260810 18:52:45.633057 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:45.645056 16742 raft_consensus.cc:3037] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:45.644295 16748 rpcz_store.cc:276] 0810 18:52:44.609269 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:44.618321 (+ 9052us) service_pool.cc:227] Handling call
0810 18:52:44.690120 (+ 71799us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:44.690355 (+ 235us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:44.690383 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:44.690853 (+ 470us) log.cc:842] Serialized 26 byte log entry
0810 18:52:45.279631 (+588778us) raft_consensus.cc:1631] Marking committed up to 6
0810 18:52:45.279686 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:45.279781 (+ 95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:45.636731 (+356950us) raft_consensus.cc:1671] finished
0810 18:52:45.636755 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:45.636883 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":511636,"threads_started":1,"wal-append.queue_time_us":881488}
I20260810 18:52:45.641907 16506 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56468 (request call id 349) took 1141 ms. Trace:
I20260810 18:52:45.646863 16506 rpcz_store.cc:276] 0810 18:52:44.500272 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:44.514112 (+ 13840us) service_pool.cc:227] Handling call
0810 18:52:44.514457 (+ 345us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:44.743722 (+229265us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:45.641685 (+897963us) spinlock_profiling.cc:243] Waited 205 ms on lock 0x7b54001e14d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:45.641847 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":472594176}
I20260810 18:52:45.580987 17727 leader_election.cc:302] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:45.641058 17770 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55586 (request call id 473) took 1897 ms. Trace:
I20260810 18:52:45.648711 17770 rpcz_store.cc:276] 0810 18:52:43.744032 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:43.761946 (+ 17914us) service_pool.cc:227] Handling call
0810 18:52:44.561389 (+799443us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:44.619342 (+ 57953us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:45.640864 (+1021522us) spinlock_profiling.cc:243] Waited 753 ms on lock 0x7b5400561b50. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:45.640999 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1730967296}
I20260810 18:52:45.619596 16448 leader_election.cc:302] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:45.621942 16631 raft_consensus.cc:2454] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 3.
I20260810 18:52:45.653290 16747 raft_consensus.cc:2454] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 8.
I20260810 18:52:45.478667 21592 raft_consensus.cc:3032] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Stepping down as leader of term 4
W20260810 18:52:45.656968 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 76104.9us
W20260810 18:52:45.600059 21705 consensus_meta.cc:287] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.722s user 0.009s sys 0.002s
W20260810 18:52:45.562287 21856 log.cc:924] Time spent T 681439c754d942b7847d73744abbe269 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20260810 18:52:45.620376 21850 raft_consensus.cc:2781] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:52:45.639461 21613 consensus_queue.cc:1035] T 4c87160448924498ac7de68947b3dc19 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
I20260810 18:52:45.673817 21888 raft_consensus.cc:2726] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:45.656868 21380 raft_consensus.cc:504] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:45.676581 21380 leader_election.cc:288] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:45.677114 21380 raft_consensus.cc:482] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:45.677356 21864 raft_consensus.cc:482] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:45.640218 21038 raft_consensus.cc:482] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:45.678175 21850 raft_consensus.cc:686] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 6 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:45.678079 21853 consensus_queue.cc:1046] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260810 18:52:45.679574 17771 raft_consensus.cc:1226] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260810 18:52:45.680135 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:45.633348 21699 consensus_meta.cc:287] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 1.167s user 0.005s sys 0.011s
W20260810 18:52:45.680624 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 61148.2us
W20260810 18:52:45.634626 21502 consensus_meta.cc:287] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb: Time spent flushing consensus metadata: real 0.529s user 0.005s sys 0.005s
I20260810 18:52:45.674940 21888 raft_consensus.cc:2726] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:52:45.596253 21685 raft_consensus.cc:482] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:45.637871 21826 log.cc:924] Time spent T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.363s user 0.000s sys 0.001s
W20260810 18:52:45.633811 21822 log.cc:924] Time spent T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.295s user 0.000s sys 0.000s
W20260810 18:52:45.558388 21817 log.cc:924] Time spent T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.343s user 0.001s sys 0.001s
I20260810 18:52:45.688200 16450 leader_election.cc:302] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:45.609292 21498 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:45.468564 21004 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:45.699797 17785 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55570 (request call id 566) took 1243 ms. Trace:
I20260810 18:52:45.685858 16742 raft_consensus.cc:2454] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 8.
I20260810 18:52:45.671603 21592 raft_consensus.cc:729] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
W20260810 18:52:45.702704 21711 consensus_meta.cc:287] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.822s user 0.006s sys 0.003s
I20260810 18:52:45.643586 21733 raft_consensus.cc:504] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:45.602238 21493 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:45.599643 21461 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:45.835773 16752 raft_consensus.cc:2454] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 8.
W20260810 18:52:45.702050 21444 consensus_meta.cc:287] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.984s user 0.012s sys 0.002s
I20260810 18:52:45.685323 21849 raft_consensus.cc:2781] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader election won for term 8
W20260810 18:52:45.991183 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 348581us
I20260810 18:52:45.697602 21546 raft_consensus.cc:3037] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 5 FOLLOWER]: Advancing to term 6
W20260810 18:52:45.729034 21821 log.cc:924] Time spent T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.430s user 0.000s sys 0.001s
I20260810 18:52:45.684043 21705 raft_consensus.cc:2726] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260810 18:52:45.702723 21837 log.cc:924] Time spent T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.230s user 0.002s sys 0.000s
I20260810 18:52:45.851148 16573 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:46.017565 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:46.029860 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 341757us
I20260810 18:52:46.003365 17785 rpcz_store.cc:276] 0810 18:52:44.456760 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:44.462762 (+ 6002us) service_pool.cc:227] Handling call
0810 18:52:44.463122 (+ 360us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:44.700193 (+237071us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:45.699560 (+999367us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b540072f6d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:45.699740 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":533587328}
W20260810 18:52:46.066582 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 215593us
I20260810 18:52:45.729864 21038 raft_consensus.cc:3037] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
W20260810 18:52:45.738960 21732 consensus_meta.cc:287] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.876s user 0.012s sys 0.000s
W20260810 18:52:46.032789 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 427210us
I20260810 18:52:46.073494 16701 leader_election.cc:302] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:45.018646 16623 rpcz_store.cc:276] 0810 18:52:43.997438 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:44.386514 (+389076us) service_pool.cc:227] Handling call
0810 18:52:44.386845 (+ 331us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:44.427217 (+ 40372us) raft_consensus.cc:1520] Early marking committed up to index 7
0810 18:52:44.515661 (+ 88444us) log.cc:842] Serialized 14 byte log entry
0810 18:52:45.008592 (+492931us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:45.008642 (+ 50us) raft_consensus.cc:1631] Marking committed up to 7
0810 18:52:45.008685 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:45.008749 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:45.011523 (+ 2774us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":69079,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":1286,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":6955,"thread_start_us":492490,"threads_started":1}
I20260810 18:52:46.074530 21732 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:46.081951 21732 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:46.083877 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:46.085074 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:46.086472 16506 raft_consensus.cc:1261] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260810 18:52:46.088802 16505 raft_consensus.cc:1226] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:45.981477 21881 raft_consensus.cc:482] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:46.043889 16504 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1710) took 1348 ms. Trace:
I20260810 18:52:46.111606 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
W20260810 18:52:45.730196 21814 log.cc:924] Time spent T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.423s user 0.000s sys 0.001s
I20260810 18:52:46.070365 21842 raft_consensus.cc:482] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:46.003006 21883 raft_consensus.cc:482] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:45.697556 21886 raft_consensus.cc:2726] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260810 18:52:45.753515 21154 raft_consensus.cc:482] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:46.132380 21154 raft_consensus.cc:504] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:46.134294 21725 raft_consensus.cc:2726] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260810 18:52:46.111078 16745 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37716 (request call id 480) took 1505 ms (client timeout 1625 ms). Trace:
W20260810 18:52:46.135285 16745 rpcz_store.cc:269] 0810 18:52:44.605607 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:44.605767 (+ 160us) service_pool.cc:227] Handling call
0810 18:52:46.110858 (+1505091us) spinlock_profiling.cc:243] Waited 301 ms on lock 0x7b54002c2ed4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:46.111000 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":692817664}
I20260810 18:52:46.139523 16745 raft_consensus.cc:1261] T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260810 18:52:46.140440 21870 raft_consensus.cc:2726] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:46.141294 16745 raft_consensus.cc:1226] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260810 18:52:45.699950 21726 raft_consensus.cc:2726] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260810 18:52:46.111980 16504 rpcz_store.cc:276] 0810 18:52:44.695456 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:44.695648 (+ 192us) service_pool.cc:227] Handling call
0810 18:52:44.745268 (+ 49620us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:45.120097 (+374829us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:46.043640 (+923543us) spinlock_profiling.cc:243] Waited 374 ms on lock 0x7b5400334354. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:46.043831 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":860534144}
I20260810 18:52:46.146013 21842 raft_consensus.cc:504] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:46.113199 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 47451.8us
W20260810 18:52:46.113093 16575 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.011s sys 0.019s
I20260810 18:52:46.146945 21493 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:46.147271 16575 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.576s user 0.011s sys 0.019s
I20260810 18:52:46.148134 21842 leader_election.cc:288] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:46.148535 21733 leader_election.cc:288] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:46.149060 21733 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:46.149533 21733 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:45.720610 21850 consensus_queue.cc:237] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:46.097364 21678 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:46.116416 16450 leader_election.cc:302] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:46.037825 16507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1712) took 1295 ms. Trace:
W20260810 18:52:46.089435 21748 consensus_meta.cc:287] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 1.278s user 0.007s sys 0.004s
W20260810 18:52:46.087050 21810 consensus_meta.cc:287] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 0.970s user 0.011s sys 0.000s
W20260810 18:52:46.096504 21296 consensus_meta.cc:287] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 1.073s user 0.007s sys 0.004s
W20260810 18:52:46.120769 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 47754.9us
I20260810 18:52:46.018597 21873 raft_consensus.cc:482] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:46.124437 16502 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52352 (request call id 1704) took 1460 ms (client timeout 1861 ms). Trace:
I20260810 18:52:46.039104 21839 raft_consensus.cc:2781] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader election won for term 8
I20260810 18:52:46.144081 16506 raft_consensus.cc:1226] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260810 18:52:46.063887 21849 raft_consensus.cc:686] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [term 8 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:46.142808 16745 raft_consensus.cc:1226] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 7. Current term is 8. Ops: []
I20260810 18:52:45.741992 21806 raft_consensus.cc:686] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 10 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:45.705307 21380 raft_consensus.cc:504] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:46.104441 21498 raft_consensus.cc:482] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:45.753371 21502 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:45.695272 21864 raft_consensus.cc:504] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:46.003306 21685 raft_consensus.cc:504] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:46.005844 21444 raft_consensus.cc:504] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:46.007589 21711 raft_consensus.cc:504] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:46.043362 21699 consensus_queue.cc:1046] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.154s
I20260810 18:52:45.878618 21874 consensus_queue.cc:1046] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.399s
I20260810 18:52:46.144925 21592 consensus_queue.cc:260] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:46.159513 16740 raft_consensus.cc:3037] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Advancing to term 9
W20260810 18:52:46.437327 16447 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.008s sys 0.016s
W20260810 18:52:46.746587 16447 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.883s user 0.011s sys 0.022s
I20260810 18:52:46.745956 21849 consensus_queue.cc:237] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:46.749584 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:52:46.750152 16631 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.597s user 0.006s sys 0.017s
I20260810 18:52:46.750254 16623 raft_consensus.cc:3037] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:46.751235 21004 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:46.750871 16752 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s user 0.002s sys 0.014s
W20260810 18:52:46.752286 16752 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.544s user 0.002s sys 0.014s
W20260810 18:52:46.757411 16631 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.605s user 0.006s sys 0.017s
I20260810 18:52:46.747263 21873 raft_consensus.cc:504] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:46.684072 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:46.288879 21739 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.743s user 0.013s sys 0.012s
I20260810 18:52:46.156701 16747 raft_consensus.cc:1226] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 6. Current term is 7. Ops: [6.8-6.8]
W20260810 18:52:46.557001 17727 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.877s user 0.012s sys 0.015s
I20260810 18:52:46.153683 17787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55602 (request call id 439) took 2085 ms. Trace:
W20260810 18:52:46.567277 16748 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.706s user 0.013s sys 0.008s
I20260810 18:52:46.351508 16750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2173) took 3229 ms. Trace:
W20260810 18:52:46.766140 17724 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.854s user 0.003s sys 0.007s
I20260810 18:52:46.766469 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:46.749967 16701 leader_election.cc:302] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:46.766455 16504 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.620s user 0.001s sys 0.009s
W20260810 18:52:46.767093 17726 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.684s user 0.001s sys 0.009s
W20260810 18:52:46.767882 16448 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.877s user 0.006s sys 0.005s
W20260810 18:52:46.768250 16573 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.701s user 0.003s sys 0.007s
W20260810 18:52:46.750859 16502 rpcz_store.cc:269] 0810 18:52:44.664551 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:44.664711 (+ 160us) service_pool.cc:227] Handling call
0810 18:52:46.123954 (+1459243us) spinlock_profiling.cc:243] Waited 12.8 ms on lock 0x7b5400146254. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:46.124362 (+ 408us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":29358976}
I20260810 18:52:46.772079 16506 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1723) took 1496 ms. Trace:
I20260810 18:52:46.772392 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:46.254386 16501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1651) took 2190 ms. Trace:
I20260810 18:52:46.763206 16507 rpcz_store.cc:276] 0810 18:52:44.742616 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:44.742754 (+ 138us) service_pool.cc:227] Handling call
0810 18:52:44.751814 (+ 9060us) raft_consensus.cc:1457] Updating replica for 2 ops
0810 18:52:45.131388 (+379574us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:45.131417 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0810 18:52:45.145183 (+ 13766us) log.cc:842] Serialized 52 byte log entry
0810 18:52:45.444394 (+299211us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:45.444430 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:45.444772 (+ 342us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:46.037628 (+592856us) raft_consensus.cc:1671] finished
0810 18:52:46.037646 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:46.037770 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":28469,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":7404,"thread_start_us":618609,"threads_started":2,"wal-append.queue_time_us":765802}
I20260810 18:52:46.773239 16623 raft_consensus.cc:2454] T 6943a9cdcb8f48dd817a90f357bf54f1 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 9.
W20260810 18:52:46.773283 16749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43726 (request call id 1635) took 1896 ms (client timeout 2236 ms). Trace:
W20260810 18:52:46.120597 21829 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.000s sys 0.020s
W20260810 18:52:46.774309 17727 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.094s user 0.016s sys 0.018s
W20260810 18:52:46.792623 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 1.11274e+06us
W20260810 18:52:46.774781 16748 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.913s user 0.016s sys 0.011s
I20260810 18:52:46.788462 16740 raft_consensus.cc:2454] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 9.
I20260810 18:52:46.775204 21748 raft_consensus.cc:504] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:46.774794 16750 rpcz_store.cc:276] 0810 18:52:43.121990 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:44.080211 (+958221us) service_pool.cc:227] Handling call
0810 18:52:44.343728 (+263517us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:44.702054 (+358326us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:44.702100 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:44.702683 (+ 583us) log.cc:842] Serialized 26 byte log entry
0810 18:52:45.137512 (+434829us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:45.137567 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:45.137659 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:45.702115 (+564456us) spinlock_profiling.cc:243] Waited 46.6 ms on lock 0x7b44000232cc. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae50e4a95 00007f1ae50e54b2 00007f1ae7d8d511 00007f1ae7d88e1a 00007f1ae7d7cbee 00007f1ae7d7bf36 00007f1ae7d7bb58 00007f1ae7d85551 00007f1ae7d85509 00007f1ae7d85499
0810 18:52:46.297174 (+595059us) raft_consensus.cc:1671] finished
0810 18:52:46.297197 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:46.351399 (+ 54202us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":588,"raft.queue_time_us":102248,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":85,"spinlock_wait_cycles":107293056,"thread_start_us":723428,"threads_started":2,"wal-append.queue_time_us":635955}
W20260810 18:52:46.567323 21598 consensus_meta.cc:287] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 1.313s user 0.012s sys 0.000s
W20260810 18:52:46.742502 21903 log.cc:924] Time spent T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.298s user 0.000s sys 0.000s
I20260810 18:52:46.889696 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:46.784301 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 667919us
I20260810 18:52:46.893585 16752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2209) took 1288 ms. Trace:
I20260810 18:52:46.893800 21502 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:46.135013 21725 consensus_queue.cc:1046] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.760s
I20260810 18:52:46.894002 16752 rpcz_store.cc:276] 0810 18:52:45.605132 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:46.031481 (+426349us) service_pool.cc:227] Handling call
0810 18:52:46.117065 (+ 85584us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:46.117499 (+ 434us) raft_consensus.cc:1520] Early marking committed up to index 3
0810 18:52:46.207708 (+ 90209us) log.cc:842] Serialized 14 byte log entry
0810 18:52:46.893232 (+685524us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:46.893287 (+ 55us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:46.893335 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:46.893401 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:46.893526 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":140503,"prepare.queue_time_us":90459,"prepare.run_cpu_time_us":105,"prepare.run_wall_time_us":7579,"raft.queue_time_us":141596,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":85,"thread_start_us":544544,"threads_started":1}
I20260810 18:52:46.895514 21699 raft_consensus.cc:2726] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:46.895648 21733 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:46.895720 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 145829us
I20260810 18:52:46.896258 21733 raft_consensus.cc:3032] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 LEADER]: Stepping down as leader of term 6
I20260810 18:52:46.896579 16701 leader_election.cc:302] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:46.896742 21733 raft_consensus.cc:729] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:46.897836 16701 leader_election.cc:302] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:46.898685 21886 raft_consensus.cc:2781] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260810 18:52:46.899127 21886 raft_consensus.cc:482] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Starting leader election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:46.899036 21874 consensus_queue.cc:1035] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:52:46.899627 21886 raft_consensus.cc:3037] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Advancing to term 10
I20260810 18:52:46.900870 21685 leader_election.cc:288] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:46.903337 21887 raft_consensus.cc:2726] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260810 18:52:46.254931 21038 raft_consensus.cc:504] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:46.907292 21038 leader_election.cc:288] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:46.907922 21896 raft_consensus.cc:482] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:46.908564 21896 raft_consensus.cc:504] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:46.911161 16742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2211) took 1226 ms. Trace:
I20260810 18:52:46.910871 21896 leader_election.cc:288] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:46.911782 21887 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:46.911566 16742 rpcz_store.cc:276] 0810 18:52:45.684579 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:46.034565 (+349986us) service_pool.cc:227] Handling call
0810 18:52:46.034997 (+ 432us) raft_consensus.cc:1457] Updating replica for 2 ops
0810 18:52:46.035583 (+ 586us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:46.035622 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0810 18:52:46.070158 (+ 34536us) log.cc:842] Serialized 52 byte log entry
0810 18:52:46.745912 (+675754us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:46.745957 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:46.746037 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:46.910905 (+164868us) raft_consensus.cc:1671] finished
0810 18:52:46.910928 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:46.911088 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":57995,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":131,"thread_start_us":467727,"threads_started":1,"wal-append.queue_time_us":817787}
I20260810 18:52:46.912811 21887 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:46.776151 16448 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.913s user 0.006s sys 0.005s
I20260810 18:52:46.913019 21806 consensus_queue.cc:237] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:46.782234 21873 leader_election.cc:288] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:46.915297 21296 raft_consensus.cc:2781] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader election won for term 8
I20260810 18:52:46.827342 17727 leader_election.cc:302] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:46.917814 16632 raft_consensus.cc:2379] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 7: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:46.922416 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:52:46.775933 17726 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.693s user 0.001s sys 0.009s
I20260810 18:52:46.776726 16506 rpcz_store.cc:276] 0810 18:52:45.275914 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:46.112827 (+836913us) service_pool.cc:227] Handling call
0810 18:52:46.143846 (+ 31019us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:46.771745 (+627899us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:46.772016 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 18:52:46.219650 21848 consensus_meta.cc:287] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.759s user 0.010s sys 0.002s
I20260810 18:52:46.780193 16507 raft_consensus.cc:1226] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 8. Current term is 9. Ops: []
I20260810 18:52:46.931769 16507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56468 (request call id 352) took 1625 ms. Trace:
W20260810 18:52:46.932150 16701 connection.cc:232] RPC call timeout handler was delayed by -1.4806551390793175s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22160000000000002s for a response
W20260810 18:52:46.767791 21828 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.837s user 0.004s sys 0.006s
W20260810 18:52:46.766867 21605 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.728s user 0.007s sys 0.003s
I20260810 18:52:46.118856 21881 raft_consensus.cc:504] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:46.888182 21598 raft_consensus.cc:2726] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260810 18:52:46.829260 21829 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.193s user 0.006s sys 0.046s
I20260810 18:52:46.900830 21874 consensus_queue.cc:1046] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 18:52:46.911638 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:52:46.517043 21808 consensus_meta.cc:287] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be: Time spent flushing consensus metadata: real 1.393s user 0.005s sys 0.010s
I20260810 18:52:46.948601 16506 raft_consensus.cc:1261] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260810 18:52:46.523742 21895 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:46.940907 21605 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.903s user 0.019s sys 0.013s
W20260810 18:52:46.775723 16504 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.629s user 0.002s sys 0.009s
W20260810 18:52:46.775457 17724 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.863s user 0.003s sys 0.007s
I20260810 18:52:46.919209 21896 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:46.916764 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:46.782441 16502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:52330 (request call id 865) took 1177 ms. Trace:
W20260810 18:52:46.947674 16443 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.019s sys 0.011s
I20260810 18:52:46.932358 16507 rpcz_store.cc:276] 0810 18:52:45.306741 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:46.778417 (+1471676us) service_pool.cc:227] Handling call
0810 18:52:46.778838 (+ 421us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:46.931421 (+152583us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:46.931694 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 18:52:46.780092 16749 rpcz_store.cc:269] 0810 18:52:44.877298 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:45.099816 (+222518us) service_pool.cc:227] Handling call
0810 18:52:46.683835 (+1584019us) spinlock_profiling.cc:243] Waited 40.6 ms on lock 0x7b5800270f98. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed03c1b0 00007f1aed03ba94 00007f1ae6723c04 00007f1ae6723b92
0810 18:52:46.773221 (+ 89386us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":93480704}
W20260810 18:52:46.773979 21739 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.245s user 0.016s sys 0.027s
I20260810 18:52:46.777806 16501 rpcz_store.cc:276] 0810 18:52:44.063935 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:44.064089 (+ 154us) service_pool.cc:227] Handling call
0810 18:52:44.064421 (+ 332us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:44.407457 (+343036us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:44.407501 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:44.411355 (+ 3854us) log.cc:842] Serialized 26 byte log entry
0810 18:52:44.953650 (+542295us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:44.953692 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:44.953889 (+ 197us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:45.562692 (+608803us) spinlock_profiling.cc:243] Waited 6.75 ms on lock 0x7b480024c258. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1aecfa17f1 00007f1aecf988e7 00007f1aecf9877a 00007f1aed08c95f 00007f1aed08c627 00007f1aed0a0b2c 00007f1aed0a0ad8 00007f1aed0a0a58 00007f1aed0a0a13 00007f1aed09fcbc 00007f1aec021be6
0810 18:52:46.225450 (+662758us) raft_consensus.cc:1671] finished
0810 18:52:46.225478 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:46.254293 (+ 28815us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":342623,"raft.queue_time_us":346797,"raft.run_cpu_time_us":5474,"raft.run_wall_time_us":1147060,"spinlock_wait_cycles":15518592,"thread_start_us":484943,"threads_started":1,"wal-append.queue_time_us":759107}
I20260810 18:52:46.774485 17787 rpcz_store.cc:276] 0810 18:52:44.068675 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:44.068830 (+ 155us) service_pool.cc:227] Handling call
0810 18:52:44.069213 (+ 383us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:44.417843 (+348630us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:44.417888 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:44.459648 (+ 41760us) log.cc:842] Serialized 26 byte log entry
0810 18:52:45.042852 (+583204us) raft_consensus.cc:1631] Marking committed up to 7
0810 18:52:45.042905 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:45.042992 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:46.153465 (+1110473us) raft_consensus.cc:1671] finished
0810 18:52:46.153484 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:46.153621 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":324524,"raft.run_cpu_time_us":50,"raft.run_wall_time_us":51,"thread_start_us":928029,"threads_started":2,"wal-append.queue_time_us":764971}
I20260810 18:52:46.939004 16505 raft_consensus.cc:1226] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:46.782974 21678 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:46.941830 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 192173us
I20260810 18:52:46.897887 21733 consensus_queue.cc:260] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:46.944159 16500 raft_consensus.cc:1226] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:46.945639 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:46.949184 16506 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1668) took 1309 ms. Trace:
I20260810 18:52:46.886641 21004 raft_consensus.cc:3032] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 6 LEADER]: Stepping down as leader of term 6
I20260810 18:52:46.949200 16743 raft_consensus.cc:1261] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260810 18:52:46.793249 21922 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:46.941041 21828 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.021s user 0.012s sys 0.021s
W20260810 18:52:46.776366 16573 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.709s user 0.003s sys 0.007s
I20260810 18:52:46.876215 21461 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:46.827780 21839 raft_consensus.cc:686] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [term 8 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:46.783478 21870 raft_consensus.cc:482] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:46.980475 21739 consensus_queue.cc:1225] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer 0397d760589d40d3b92ef141cdfc2dbb log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260810 18:52:46.955555 16502 rpcz_store.cc:276] 0810 18:52:45.605246 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:46.780625 (+1175379us) service_pool.cc:227] Handling call
0810 18:52:46.782332 (+ 1707us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:46.917315 21873 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
W20260810 18:52:47.034732 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 142024us
W20260810 18:52:47.036479 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 873641us
I20260810 18:52:46.919582 21887 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:46.957882 16623 raft_consensus.cc:3037] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:46.961766 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:46.962004 21901 raft_consensus.cc:482] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:46.962745 21881 leader_election.cc:288] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:46.963193 16750 raft_consensus.cc:1261] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260810 18:52:46.968279 16622 raft_consensus.cc:1261] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260810 18:52:47.097631 16622 pending_rounds.cc:76] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb: Aborting all ops after (but not including) 2
I20260810 18:52:47.097864 16622 pending_rounds.cc:97] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260810 18:52:47.098141 16622 raft_consensus.cc:2866] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:52:46.982033 16501 raft_consensus.cc:1226] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 8. Current term is 9. Ops: []
I20260810 18:52:46.980865 16506 rpcz_store.cc:276] 0810 18:52:45.639593 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:46.929992 (+1290399us) service_pool.cc:227] Handling call
0810 18:52:46.948313 (+ 18321us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:46.948979 (+ 666us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:46.949137 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:46.942567 21912 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:46.783221 21498 raft_consensus.cc:504] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } }
W20260810 18:52:47.015339 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 60477.8us
I20260810 18:52:47.017269 21598 raft_consensus.cc:482] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
W20260810 18:52:47.021044 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 953614us
I20260810 18:52:46.981385 21896 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:47.012957 21004 raft_consensus.cc:729] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:46.983526 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:47.114004 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:46.983883 16507 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454"
dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
mode: GRACEFUL
new_leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
from {username='slave'} at 127.0.0.1:51572
I20260810 18:52:47.024775 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:46.955392 16626 raft_consensus.cc:3037] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:47.005888 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:47.030330 16508 raft_consensus.cc:1226] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [term 6 LEADER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6. Ops: []
W20260810 18:52:47.030664 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 203409us
I20260810 18:52:47.117419 17727 leader_election.cc:302] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:47.117741 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:47.118160 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:47.118350 16749 raft_consensus.cc:2454] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:46.154009 21883 raft_consensus.cc:504] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:46.942376 21848 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:46.955761 16443 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.160s user 0.020s sys 0.011s
I20260810 18:52:46.983282 21924 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:47.067237 16502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:52330 (request call id 864) took 1586 ms. Trace:
I20260810 18:52:47.125746 16502 rpcz_store.cc:276] 0810 18:52:45.451037 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:47.036501 (+1585464us) service_pool.cc:227] Handling call
0810 18:52:47.037716 (+ 1215us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:46.952785 21808 raft_consensus.cc:504] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:47.073890 16504 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1724) took 1797 ms. Trace:
I20260810 18:52:47.114064 21598 raft_consensus.cc:504] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:47.127130 16504 rpcz_store.cc:276] 0810 18:52:45.276183 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:46.145332 (+869149us) service_pool.cc:227] Handling call
0810 18:52:46.145722 (+ 390us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:46.145971 (+ 249us) raft_consensus.cc:1520] Early marking committed up to index 8
0810 18:52:46.146391 (+ 420us) log.cc:842] Serialized 14 byte log entry
0810 18:52:47.073545 (+927154us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:47.073599 (+ 54us) raft_consensus.cc:1631] Marking committed up to 8
0810 18:52:47.073647 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:47.073713 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:47.073834 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10871,"prepare.queue_time_us":2553,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":136220,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":13141,"spinlock_wait_cycles":71749888,"thread_start_us":922609,"threads_started":2,"wal-append.queue_time_us":794987}
I20260810 18:52:47.033428 21853 raft_consensus.cc:2781] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader election won for term 8
I20260810 18:52:47.128394 21461 consensus_queue.cc:1046] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260810 18:52:47.075871 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:47.129240 21598 leader_election.cc:288] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:47.129871 21881 raft_consensus.cc:3032] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 LEADER]: Stepping down as leader of term 7
I20260810 18:52:47.130381 21881 raft_consensus.cc:729] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:47.131462 21380 leader_election.cc:288] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:47.131412 21881 consensus_queue.cc:260] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:47.132555 21864 leader_election.cc:288] T 50fbe8baa0b345b0b2c4e4489e397660 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:46.951464 21914 raft_consensus.cc:482] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:47.133507 21546 consensus_meta.cc:287] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 1.101s user 0.006s sys 0.007s
I20260810 18:52:47.133468 21914 raft_consensus.cc:504] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:47.134429 17775 raft_consensus.cc:2454] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
W20260810 18:52:47.135736 21925 log.cc:924] Time spent T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
I20260810 18:52:47.137143 16748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2203) took 1858 ms. Trace:
I20260810 18:52:47.112588 21839 consensus_queue.cc:237] T 9dc14894db5b4872b0a5dd8d933426f4 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:47.137517 16748 rpcz_store.cc:276] 0810 18:52:45.279153 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:45.655658 (+376505us) service_pool.cc:227] Handling call
0810 18:52:45.692728 (+ 37070us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:45.709657 (+ 16929us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:45.709702 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:45.729091 (+ 19389us) log.cc:842] Serialized 26 byte log entry
0810 18:52:46.793533 (+1064442us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:46.793592 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:46.793686 (+ 94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:47.136931 (+343245us) raft_consensus.cc:1671] finished
0810 18:52:47.136949 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:47.137085 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28318,"raft.queue_time_us":51171,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":41879,"thread_start_us":932060,"threads_started":1,"wal-append.queue_time_us":1268596}
I20260810 18:52:47.139349 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:47.077750 21705 raft_consensus.cc:482] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:47.140502 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:47.084801 21942 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:47.035247 21870 raft_consensus.cc:504] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:47.141158 21942 raft_consensus.cc:504] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:47.141765 21711 leader_election.cc:288] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:47.092700 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:47.143527 16742 raft_consensus.cc:2354] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6.
I20260810 18:52:46.957230 21895 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:47.099145 16501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1670) took 1459 ms. Trace:
I20260810 18:52:47.105741 21359 consensus_queue.cc:1046] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260810 18:52:47.076015 21909 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:47.217127 21909 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:47.024822 21922 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:47.220744 21922 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:47.119375 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:47.221818 21886 raft_consensus.cc:504] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:47.222285 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:47.125365 21444 leader_election.cc:288] T 6fa467f263e640b095b31b6d998258db P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:47.223907 21895 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:47.035988 21873 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:47.121452 16506 raft_consensus.cc:1226] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:47.122157 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:47.107319 21549 raft_consensus.cc:3032] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 LEADER]: Stepping down as leader of term 6
I20260810 18:52:47.138357 21803 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:47.140304 21705 raft_consensus.cc:504] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:47.082410 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 46307.7us
I20260810 18:52:47.148216 16748 raft_consensus.cc:2454] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:47.239476 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:47.080039 17726 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:47.111161 21296 raft_consensus.cc:686] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [term 8 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:47.112993 21910 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:47.111354 21955 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:47.109606 21943 raft_consensus.cc:482] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:47.208285 16575 leader_election.cc:302] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:47.208765 16495 raft_consensus.cc:1226] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 7. Current term is 8. Ops: []
W20260810 18:52:47.195541 16701 leader_election.cc:334] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.216s (SENT)
I20260810 18:52:47.257494 16701 leader_election.cc:302] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:47.257565 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 49380.8us
I20260810 18:52:47.196658 16623 raft_consensus.cc:2454] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 8.
I20260810 18:52:47.152463 16501 rpcz_store.cc:276] 0810 18:52:45.640071 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:46.968663 (+1328592us) service_pool.cc:227] Handling call
0810 18:52:46.969303 (+ 640us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:47.098800 (+129497us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:47.099053 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 18:52:47.190387 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 43788.6us
I20260810 18:52:47.223017 16749 raft_consensus.cc:2354] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 7. Current term is 8.
I20260810 18:52:47.258493 16507 raft_consensus.cc:595] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Received request to transfer leadership to TS 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:47.120028 16695 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:47.258946 16507 raft_consensus.cc:603] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260810 18:52:47.259186 16575 leader_election.cc:302] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters: d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:47.136574 21914 leader_election.cc:288] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:47.146394 16750 raft_consensus.cc:2454] T c3b33c2c321842498975ee1086991616 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:47.224763 21848 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:47.273043 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:47.273356 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 105190us
I20260810 18:52:47.273861 16749 raft_consensus.cc:2454] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
I20260810 18:52:47.240378 16748 raft_consensus.cc:2362] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 8. Re-sending same reply.
I20260810 18:52:47.152168 21870 leader_election.cc:288] T 3e58f8d2231b4e2d900ec099ae700d09 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:47.135002 21808 leader_election.cc:288] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:47.256040 16626 raft_consensus.cc:2454] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 6.
I20260810 18:52:47.189080 21748 leader_election.cc:288] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:47.260560 16575 leader_election.cc:302] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:47.266045 16323 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 8810b6b0b676405daf5ffddc50a4d21f with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica d304ac47fdf846de81337057a4ef7bea is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 9 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:52:47.143942 21557 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:47.272600 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:47.274700 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:47.293754 16749 raft_consensus.cc:2396] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260810 18:52:47.276057 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:47.242718 21803 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:47.272137 21943 raft_consensus.cc:504] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:47.277675 21845 consensus_queue.cc:1035] T 4e623fc5613147efbb77f01c3b0b7e81 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:52:47.279264 21895 raft_consensus.cc:2781] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:47.228837 21549 raft_consensus.cc:729] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
W20260810 18:52:47.281457 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 161502us
I20260810 18:52:47.229421 21546 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:47.099268 21912 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:47.123152 21937 consensus_queue.cc:1046] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.169s
I20260810 18:52:47.111112 21901 raft_consensus.cc:504] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:47.151729 21924 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:47.246762 21004 consensus_queue.cc:260] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:47.300678 21678 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:47.473295 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:47.326321 21557 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:47.248857 21881 raft_consensus.cc:3037] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:47.491362 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:47.492115 16752 raft_consensus.cc:3037] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Advancing to term 9
I20260810 18:52:47.477247 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:47.473188 16450 leader_election.cc:302] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:47.261365 21910 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:47.500659 21910 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:47.504032 21943 leader_election.cc:288] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:47.505890 16496 raft_consensus.cc:3037] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:47.268093 21490 raft_consensus.cc:2726] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:47.507809 21896 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:47.508780 21803 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:47.287424 21598 raft_consensus.cc:2726] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260810 18:52:47.290426 21461 raft_consensus.cc:482] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:47.509976 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:47.510681 21498 leader_election.cc:288] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:47.371803 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:47.495172 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:47.291258 21502 consensus_queue.cc:1035] T dac7251004484853a9326a55ffbb9c6d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.206s
I20260810 18:52:47.371739 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:52:47.412081 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 332164us
I20260810 18:52:47.272045 21950 raft_consensus.cc:2726] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260810 18:52:47.522630 17726 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:52:47.519904 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 259410us
I20260810 18:52:47.530791 17726 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:47.524307 16749 raft_consensus.cc:2379] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 8: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:47.530706 16575 leader_election.cc:302] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:47.533473 17726 leader_election.cc:302] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:47.534802 17785 raft_consensus.cc:2454] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 8.
W20260810 18:52:47.537097 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 63992us
I20260810 18:52:47.300858 21887 raft_consensus.cc:3037] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:47.541888 16748 raft_consensus.cc:3037] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:47.553601 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:47.468736 21873 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:47.543397 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 242543us
I20260810 18:52:47.457041 21909 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:47.288524 21942 leader_election.cc:288] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:47.541877 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:47.268168 21955 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:47.372603 21849 raft_consensus.cc:2726] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260810 18:52:47.553860 21808 raft_consensus.cc:482] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:47.466841 21946 consensus_queue.cc:1046] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.162s
I20260810 18:52:47.512665 21886 leader_election.cc:288] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 10 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:47.521804 21739 raft_consensus.cc:482] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:47.281417 21380 consensus_queue.cc:1035] T 8547cf266dd24ff289d7d8ff5e2174d0 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.255s
I20260810 18:52:47.301128 21895 raft_consensus.cc:482] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:47.563874 17771 raft_consensus.cc:1261] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260810 18:52:47.574795 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:47.577129 16748 raft_consensus.cc:2454] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 9.
I20260810 18:52:47.578969 21557 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:47.542928 21004 raft_consensus.cc:3037] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:47.542060 21848 raft_consensus.cc:2781] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:47.622685 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:47.623370 16575 leader_election.cc:302] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:47.552871 21950 raft_consensus.cc:2781] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:52:47.528132 21977 raft_consensus.cc:2726] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:47.540629 21795 raft_consensus.cc:3037] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:47.533006 21605 raft_consensus.cc:2726] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260810 18:52:47.509969 21461 raft_consensus.cc:504] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:47.608151 16502 raft_consensus.cc:2396] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for term 9 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260810 18:52:47.580075 17787 raft_consensus.cc:2454] T ab9362a1df854c4fad0204ee2f9f78ad P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 9.
I20260810 18:52:47.527374 21546 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:47.587121 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:47.611493 21986 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:47.608779 16496 raft_consensus.cc:2454] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:47.468936 21915 consensus_queue.cc:1035] T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.240s
I20260810 18:52:47.588438 16752 raft_consensus.cc:2454] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 9.
I20260810 18:52:47.614295 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:47.614202 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:47.557252 21498 raft_consensus.cc:2726] T 32567b0896594c1186a56d27e36ca0a0 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260810 18:52:47.625197 16504 raft_consensus.cc:1261] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260810 18:52:47.532786 21678 consensus_queue.cc:1046] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.077s
I20260810 18:52:47.456688 21296 consensus_queue.cc:237] T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:47.577620 21154 leader_election.cc:288] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:47.610070 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:47.622344 21848 raft_consensus.cc:482] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260810 18:52:47.611225 21866 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.587s user 0.039s sys 0.039s
I20260810 18:52:47.551393 21598 consensus_queue.cc:1046] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.073s
I20260810 18:52:47.469074 21952 consensus_queue.cc:1046] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.381s
I20260810 18:52:47.555451 21896 consensus_queue.cc:1046] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.076s
I20260810 18:52:47.636247 21490 consensus_queue.cc:1035] T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.118s
I20260810 18:52:47.606521 21808 raft_consensus.cc:504] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:47.615263 21922 consensus_queue.cc:1046] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.132s
I20260810 18:52:47.886526 16626 raft_consensus.cc:1261] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 11 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260810 18:52:47.805655 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 476653us
I20260810 18:52:47.610095 21895 raft_consensus.cc:3037] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:47.888281 16504 pending_rounds.cc:76] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea: Aborting all ops after (but not including) 7
I20260810 18:52:47.727603 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:47.888801 16504 pending_rounds.cc:97] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260810 18:52:47.668607 16749 raft_consensus.cc:2454] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 9.
I20260810 18:52:47.891993 16501 raft_consensus.cc:2454] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 9.
I20260810 18:52:47.892138 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:47.736513 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:47.893754 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
W20260810 18:52:47.888010 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 264699us
W20260810 18:52:47.668963 16496 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52352 (request call id 1719) took 2874 ms (client timeout 1851 ms). Trace:
I20260810 18:52:47.892426 16450 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:47.894999 16575 leader_election.cc:302] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:47.897855 16504 raft_consensus.cc:2866] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:52:47.755597 17783 raft_consensus.cc:2454] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 9.
W20260810 18:52:47.895056 16496 rpcz_store.cc:269] 0810 18:52:44.794586 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:44.794680 (+ 94us) service_pool.cc:227] Handling call
0810 18:52:47.505397 (+2710717us) spinlock_profiling.cc:243] Waited 2.62 s on lock 0x7b58003ec0b0. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae8075f63 00007f1ae80d9aba 00007f1ae80f13af 00007f1ae80f2b4e 00007f1ae80fe228 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c
0810 18:52:47.668859 (+163462us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6027961088}
I20260810 18:52:47.909961 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:47.787674 21964 log.cc:924] Time spent T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.163s user 0.000s sys 0.001s
I20260810 18:52:47.935523 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
W20260810 18:52:47.766624 21965 log.cc:924] Time spent T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.138s user 0.000s sys 0.000s
I20260810 18:52:47.940812 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:47.941448 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:47.917953 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:47.943536 21498 raft_consensus.cc:2726] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:47.615098 21739 raft_consensus.cc:504] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:47.948740 21739 leader_election.cc:288] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:47.735201 21873 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:47.951370 21873 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:47.951884 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:47.952399 21808 leader_election.cc:288] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:47.952610 16501 raft_consensus.cc:2454] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 7.
I20260810 18:52:47.914763 16450 leader_election.cc:302] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:47.953621 21873 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:47.953914 21546 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:47.953877 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
W20260810 18:52:47.882614 21866 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.863s user 0.044s sys 0.051s
W20260810 18:52:47.939977 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 45065.3us
I20260810 18:52:47.849377 21810 raft_consensus.cc:482] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:47.667807 21883 leader_election.cc:288] T dcd14258d5d24692a3377387cb5c204e P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:47.747408 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:47.886610 21848 raft_consensus.cc:3037] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:47.668815 21038 raft_consensus.cc:482] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:47.789849 21955 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:47.955220 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:47.975123 16623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 2194) took 1071 ms. Trace:
I20260810 18:52:47.975325 16752 raft_consensus.cc:2379] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 8: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:47.975621 16623 rpcz_store.cc:276] 0810 18:52:46.903265 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:46.922045 (+ 18780us) service_pool.cc:227] Handling call
0810 18:52:47.974922 (+1052877us) spinlock_profiling.cc:243] Waited 30.8 ms on lock 0x7b5400173ed4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:47.975042 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":70776192}
I20260810 18:52:47.897254 21986 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:47.935732 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:47.735821 21502 raft_consensus.cc:2726] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260810 18:52:47.855226 21733 raft_consensus.cc:3037] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:47.919741 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:47.960388 16575 leader_election.cc:302] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:47.960570 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:47.630995 21912 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:47.626497 21950 raft_consensus.cc:482] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:47.809880 21874 raft_consensus.cc:3032] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Stepping down as leader of term 7
I20260810 18:52:47.792428 21924 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:47.963675 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 49029.6us
I20260810 18:52:47.942668 21444 raft_consensus.cc:2781] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260810 18:52:47.965104 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:47.958577 16496 raft_consensus.cc:2454] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
I20260810 18:52:47.764317 21886 raft_consensus.cc:482] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:47.954761 21518 raft_consensus.cc:3037] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:47.971908 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f642e61a7e3c45de883c78d5e5c49712" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:47.972898 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:47.971927 16448 leader_election.cc:302] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:47.968396 21887 raft_consensus.cc:2726] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:47.850291 21901 leader_election.cc:288] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:47.613775 21870 consensus_queue.cc:1046] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.348s
I20260810 18:52:47.669755 21914 raft_consensus.cc:482] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:47.987888 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:48.512115 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:48.513586 16502 raft_consensus.cc:2379] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 7: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:48.049788 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:48.515329 16626 raft_consensus.cc:2454] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:48.517091 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:47.958156 21952 consensus_queue.cc:1225] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer 831925dd8b644ebab7eb0d672a94eddd log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260810 18:52:48.518517 16506 raft_consensus.cc:2379] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 9: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:48.462842 16500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1706) took 3795 ms. Trace:
I20260810 18:52:47.996907 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:47.986186 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:48.531776 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 571428us
I20260810 18:52:48.599510 16575 leader_election.cc:302] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:48.589254 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:48.546339 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:48.531424 16749 raft_consensus.cc:2454] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
I20260810 18:52:48.601182 16752 raft_consensus.cc:2454] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 8.
I20260810 18:52:48.605551 16575 leader_election.cc:302] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:48.607514 16575 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:48.608450 16633 raft_consensus.cc:2379] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 6: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:48.610869 17786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:47.971390 21948 raft_consensus.cc:2781] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Leader election won for term 9
I20260810 18:52:48.612982 16575 leader_election.cc:398] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 election: Vote denied by peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171) with higher term. Message: Invalid argument: T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for earlier term 7. Current term is 8.
I20260810 18:52:48.629230 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:48.000597 21739 raft_consensus.cc:482] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:48.507170 21828 raft_consensus.cc:482] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:48.605994 21170 raft_consensus.cc:2781] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:48.373667 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:48.048302 21895 raft_consensus.cc:504] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:48.526844 21444 raft_consensus.cc:482] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:48.630820 16752 raft_consensus.cc:2454] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 5.
I20260810 18:52:48.530864 21950 raft_consensus.cc:3037] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:48.531136 21874 raft_consensus.cc:729] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:48.063170 21897 raft_consensus.cc:482] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:47.997160 21808 raft_consensus.cc:482] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:48.590095 21914 raft_consensus.cc:504] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:48.588820 21912 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:48.622466 21502 raft_consensus.cc:2726] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260810 18:52:48.659483 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9362a1df854c4fad0204ee2f9f78ad" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:48.048863 21598 raft_consensus.cc:482] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:48.613180 21892 raft_consensus.cc:2726] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 18:52:48.698580 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd14258d5d24692a3377387cb5c204e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:47.971215 21810 raft_consensus.cc:504] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:48.622808 21870 raft_consensus.cc:482] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:48.629979 21890 raft_consensus.cc:2781] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 18:52:48.652094 21739 raft_consensus.cc:504] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:48.657672 21828 raft_consensus.cc:504] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:48.700316 21887 raft_consensus.cc:482] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:48.668413 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:48.670257 16752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43738 (request call id 2219) took 1404 ms. Trace:
W20260810 18:52:48.641494 21795 consensus_meta.cc:287] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd: Time spent flushing consensus metadata: real 0.915s user 0.010s sys 0.001s
I20260810 18:52:47.986430 21038 raft_consensus.cc:504] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:48.704710 22003 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:48.667083 21004 consensus_meta.cc:287] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 1.041s user 0.011s sys 0.001s
I20260810 18:52:48.527370 21886 raft_consensus.cc:504] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:48.668982 21895 leader_election.cc:288] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:48.697576 21912 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:48.656812 21170 raft_consensus.cc:482] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:48.657358 21685 raft_consensus.cc:482] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:48.693472 16701 leader_election.cc:334] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.718s (SENT)
W20260810 18:52:48.741834 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 48429us
I20260810 18:52:48.673197 21444 raft_consensus.cc:3037] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Advancing to term 10
I20260810 18:52:48.635483 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:48.705564 21890 raft_consensus.cc:482] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:48.717346 21795 raft_consensus.cc:2726] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260810 18:52:48.712721 21998 log.cc:924] Time spent T b4082c2e6a774acfa97ec268831748a5 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260810 18:52:48.697209 21598 raft_consensus.cc:504] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:48.638355 21901 raft_consensus.cc:482] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 831925dd8b644ebab7eb0d672a94eddd)
I20260810 18:52:48.715262 16752 rpcz_store.cc:276] 0810 18:52:47.265396 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:48.628751 (+1363355us) service_pool.cc:227] Handling call
0810 18:52:48.670126 (+ 41375us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:48.741616 21170 raft_consensus.cc:3037] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:48.694280 21808 raft_consensus.cc:504] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:48.747913 21685 raft_consensus.cc:504] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:48.705672 21870 raft_consensus.cc:504] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:48.696048 21897 raft_consensus.cc:504] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:48.002981 21883 raft_consensus.cc:482] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:48.715926 21887 raft_consensus.cc:504] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:48.791564 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:48.771930 21890 raft_consensus.cc:3037] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:48.707787 21874 consensus_queue.cc:260] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:48.772197 17780 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55586 (request call id 492) took 1023 ms. Trace:
I20260810 18:52:48.796949 16752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43738 (request call id 2222) took 1530 ms. Trace:
I20260810 18:52:48.806686 16752 rpcz_store.cc:276] 0810 18:52:47.266330 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:48.791234 (+1524904us) service_pool.cc:227] Handling call
0810 18:52:48.796869 (+ 5635us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:48.807953 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6943a9cdcb8f48dd817a90f357bf54f1" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:48.809360 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:48.720593 22003 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:48.826545 21870 leader_election.cc:288] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:48.818486 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:48.832849 16752 raft_consensus.cc:2454] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 9.
I20260810 18:52:48.834120 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:48.835114 16752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37716 (request call id 494) took 1327 ms. Trace:
I20260810 18:52:48.835561 16752 rpcz_store.cc:276] 0810 18:52:47.507565 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:48.833784 (+1326219us) service_pool.cc:227] Handling call
0810 18:52:48.835036 (+ 1252us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.108146 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:48.535512 16500 rpcz_store.cc:276] 0810 18:52:44.667605 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:44.667769 (+ 164us) service_pool.cc:227] Handling call
0810 18:52:44.704091 (+ 36322us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:46.980591 (+2276500us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:48.462511 (+1481920us) spinlock_profiling.cc:243] Waited 2.24 s on lock 0x7b5400116054. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:48.462761 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5151242624}
I20260810 18:52:49.208264 16501 raft_consensus.cc:2454] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 8.
I20260810 18:52:49.208809 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:49.209466 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c87160448924498ac7de68947b3dc19" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:49.210367 16500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52338 (request call id 1674) took 1603 ms. Trace:
I20260810 18:52:49.116981 16496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52338 (request call id 1685) took 1202 ms. Trace:
I20260810 18:52:49.117591 16504 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56468 (request call id 356) took 1557 ms. Trace:
I20260810 18:52:48.545584 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:49.120339 16505 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1722) took 3844 ms. Trace:
W20260810 18:52:49.211360 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 1.2047e+06us
I20260810 18:52:49.211716 16505 rpcz_store.cc:276] 0810 18:52:45.275660 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:46.110696 (+835036us) service_pool.cc:227] Handling call
0810 18:52:46.153232 (+ 42536us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:46.968204 (+814972us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.119746 (+2151542us) spinlock_profiling.cc:243] Waited 785 ms on lock 0x7b54000f5d54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:49.120129 (+ 383us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1806479616}
I20260810 18:52:49.213588 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:49.214656 16505 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52338 (request call id 1682) took 1361 ms. Trace:
I20260810 18:52:49.120503 16495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1665) took 3897 ms. Trace:
I20260810 18:52:49.121264 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:49.123591 16508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1659) took 4052 ms. Trace:
W20260810 18:52:48.664556 16573 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.035s user 0.007s sys 0.007s
I20260810 18:52:48.666299 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:49.129735 16621 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1733) took 4703 ms (client timeout 2216 ms). Trace:
I20260810 18:52:49.130820 16626 raft_consensus.cc:2454] T dcd14258d5d24692a3377387cb5c204e P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:49.131950 16622 raft_consensus.cc:2454] T aa5164e185b84d0a9bf72af61bf78f84 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
W20260810 18:52:49.135622 16695 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.609s user 0.003s sys 0.011s
W20260810 18:52:48.905280 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 161548us
W20260810 18:52:49.136257 16702 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.615s user 0.010s sys 0.008s
I20260810 18:52:49.138882 16743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43726 (request call id 1637) took 4183 ms. Trace:
W20260810 18:52:49.139803 16740 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43738 (request call id 2198) took 4299 ms (client timeout 2238 ms). Trace:
I20260810 18:52:49.020116 21170 raft_consensus.cc:504] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:49.221976 16502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52338 (request call id 1679) took 1552 ms. Trace:
I20260810 18:52:49.211223 16496 rpcz_store.cc:276] 0810 18:52:47.914295 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:48.516768 (+602473us) service_pool.cc:227] Handling call
0810 18:52:49.116713 (+599945us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.215222 16495 rpcz_store.cc:276] 0810 18:52:45.223485 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:45.473930 (+250445us) service_pool.cc:227] Handling call
0810 18:52:45.527616 (+ 53686us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:47.257169 (+1729553us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.120015 (+1862846us) spinlock_profiling.cc:243] Waited 1.68 s on lock 0x7b5400117454. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:49.120344 (+ 329us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3865854208}
I20260810 18:52:49.224108 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:49.033203 21444 raft_consensus.cc:504] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:48.891920 21598 leader_election.cc:288] T 8547cf266dd24ff289d7d8ff5e2174d0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:48.842401 21685 leader_election.cc:288] T c758b1e1139941a9a8c4e703cc058061 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:49.225085 16495 raft_consensus.cc:2454] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 9.
I20260810 18:52:49.216013 16508 rpcz_store.cc:276] 0810 18:52:45.071335 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:45.377533 (+306198us) service_pool.cc:227] Handling call
0810 18:52:46.124089 (+746556us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:47.119530 (+995441us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.122227 (+2002697us) spinlock_profiling.cc:243] Waited 904 ms on lock 0x7b5400146254. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:49.123108 (+ 881us) spinlock_profiling.cc:243] Waited 746 ms on lock 0x7b5400146250. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:49.123362 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3794626176}
I20260810 18:52:48.844177 21739 leader_election.cc:288] T 33bce85658374382881d7f8d510b407d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:48.886322 21808 leader_election.cc:288] T 156c7477728b4962a7069618cdf12b92 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:48.805294 17780 rpcz_store.cc:276] 0810 18:52:47.749003 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:48.668064 (+919061us) service_pool.cc:227] Handling call
0810 18:52:48.769777 (+101713us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:48.857502 21828 leader_election.cc:288] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:48.859769 21848 consensus_meta.cc:287] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.871s user 0.012s sys 0.001s
I20260810 18:52:49.218788 16743 rpcz_store.cc:276] 0810 18:52:44.955327 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:45.437160 (+481833us) service_pool.cc:227] Handling call
0810 18:52:45.437490 (+ 330us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:47.012717 (+1575227us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.138111 (+2125394us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b540025fdd4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:49.138554 (+ 443us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3475615488}
I20260810 18:52:49.232892 21848 raft_consensus.cc:504] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:48.895884 21872 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:48.820025 21874 raft_consensus.cc:3037] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:49.235946 21872 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:49.217267 16623 raft_consensus.cc:2454] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 9.
W20260810 18:52:49.238402 17783 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55570 (request call id 578) took 1685 ms (client timeout 2208 ms). Trace:
I20260810 18:52:49.238569 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:48.804260 21883 raft_consensus.cc:504] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:48.904206 21886 leader_election.cc:288] T 0a481385212f4e189ea5b2f5e215be4d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:48.864377 21887 leader_election.cc:288] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:48.924326 21890 raft_consensus.cc:504] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:48.845850 21897 leader_election.cc:288] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:48.793736 21901 raft_consensus.cc:504] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:48.850906 21912 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:48.855829 21914 leader_election.cc:288] T 1cb6043f3f9f4b38b58e608973ac019c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:49.019191 21948 raft_consensus.cc:686] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 9 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:48.884147 21950 raft_consensus.cc:504] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:48.867826 22002 log.cc:924] Time spent T 1cb6043f3f9f4b38b58e608973ac019c P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.159s user 0.000s sys 0.001s
I20260810 18:52:48.866621 22003 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:48.856702 22004 log.cc:924] Time spent T 33bce85658374382881d7f8d510b407d P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.148s user 0.000s sys 0.001s
I20260810 18:52:49.135991 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:52:49.216452 16573 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.587s user 0.007s sys 0.008s
I20260810 18:52:49.197047 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58f8d2231b4e2d900ec099ae700d09" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:49.227308 16508 raft_consensus.cc:1261] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260810 18:52:49.218897 16740 rpcz_store.cc:269] 0810 18:52:44.840358 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:45.061791 (+221433us) service_pool.cc:227] Handling call
0810 18:52:46.111448 (+1049657us) spinlock_profiling.cc:243] Waited 57.5 ms on lock 0x7b5800270f98. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519a702 00007f1aee08ab09 00007f1aecfc6730 00007f1aed08c79b 00007f1aed08c821 00007f1aed03c1b0 00007f1aed03ba94 00007f1ae6723c04 00007f1ae6723b92
0810 18:52:49.139226 (+3027778us) spinlock_profiling.cc:243] Waited 47.1 ms on lock 0x7b540007f3d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:49.139483 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":240470656}
W20260810 18:52:49.199779 16575 leader_election.cc:334] T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Timed out: RequestConsensusVote RPC to 127.14.32.195:40171 timed out after 1.591s (SENT)
I20260810 18:52:49.200333 21518 raft_consensus.cc:482] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:49.202935 17727 leader_election.cc:302] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:49.209076 16501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52352 (request call id 1732) took 1598 ms. Trace:
I20260810 18:52:49.115788 16507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:51572 (request call id 21) took 2180 ms. Trace:
I20260810 18:52:49.210697 16500 rpcz_store.cc:276] 0810 18:52:47.607014 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.208979 (+1601965us) service_pool.cc:227] Handling call
0810 18:52:49.210271 (+ 1292us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.215034 16505 rpcz_store.cc:276] 0810 18:52:47.853540 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.213101 (+1359561us) service_pool.cc:227] Handling call
0810 18:52:49.214590 (+ 1489us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 18:52:49.218004 16695 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.692s user 0.004s sys 0.011s
W20260810 18:52:49.223026 21928 log.cc:924] Time spent T 3e58f8d2231b4e2d900ec099ae700d09 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 2.476s user 0.005s sys 0.039s
I20260810 18:52:49.224610 16496 raft_consensus.cc:1261] T 8810b6b0b676405daf5ffddc50a4d21f P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260810 18:52:49.243263 16752 raft_consensus.cc:1226] T e6768c6bbdc345eeb86337a837cc1d22 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 8. Current term is 9. Ops: []
I20260810 18:52:49.215857 16506 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52352 (request call id 1733) took 1604 ms. Trace:
I20260810 18:52:49.266637 16740 raft_consensus.cc:1261] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20260810 18:52:49.227245 16702 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.706s user 0.010s sys 0.008s
I20260810 18:52:49.222501 16502 rpcz_store.cc:276] 0810 18:52:47.669360 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:48.545230 (+875870us) service_pool.cc:227] Handling call
0810 18:52:49.221730 (+676500us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.258617 16743 raft_consensus.cc:1261] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260810 18:52:49.259274 16507 rpcz_store.cc:276] 0810 18:52:46.934787 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:46.966674 (+ 31887us) service_pool.cc:227] Handling call
0810 18:52:49.115278 (+2148604us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b5400117454. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f4bc3 00007f1aed03e81a 00007f1ae6731d04 00007f1ae6731c92 00007f1ae6731bc1 00007f1ae6731b3c 00007f1ae6730df2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:49.115528 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":328890752}
I20260810 18:52:49.272146 16743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43726 (request call id 1652) took 2312 ms. Trace:
W20260810 18:52:49.272709 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 73213.8us
I20260810 18:52:49.273104 16502 raft_consensus.cc:1261] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 9. (term mismatch)
I20260810 18:52:49.273548 16502 pending_rounds.cc:76] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea: Aborting all ops after (but not including) 8
W20260810 18:52:49.273713 16575 leader_election.cc:334] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Timed out: RequestConsensusVote RPC to 127.14.32.195:40171 timed out after 1.824s (SENT)
I20260810 18:52:49.273969 16502 pending_rounds.cc:97] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea: Aborting uncommitted NO_OP operation due to leader change: 7.9
I20260810 18:52:49.274392 16502 raft_consensus.cc:2866] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:52:49.274363 16575 leader_election.cc:302] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:49.275676 21950 leader_election.cc:288] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:49.276062 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:49.276329 21518 raft_consensus.cc:504] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:49.276960 16628 raft_consensus.cc:2454] T 91b6af1458cd44d8ba26b43579613ecb P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 3.
I20260810 18:52:49.211306 16504 rpcz_store.cc:276] 0810 18:52:47.560159 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:47.560599 (+ 440us) service_pool.cc:227] Handling call
0810 18:52:47.610016 (+ 49417us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:47.898564 (+288548us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.117035 (+1218471us) spinlock_profiling.cc:243] Waited 14.7 ms on lock 0x7b54000e1ad4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:49.117404 (+ 369us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":33890176}
W20260810 18:52:49.217437 16621 rpcz_store.cc:269] 0810 18:52:44.425973 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:44.470511 (+ 44538us) service_pool.cc:227] Handling call
0810 18:52:49.129154 (+4658643us) spinlock_profiling.cc:243] Waited 1.68 s on lock 0x7b54003845d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:49.129425 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3860925696}
I20260810 18:52:49.241845 16745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2216) took 3172 ms. Trace:
W20260810 18:52:49.238874 17783 rpcz_store.cc:269] 0810 18:52:47.552922 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:47.553251 (+ 329us) service_pool.cc:227] Handling call
0810 18:52:49.237902 (+1684651us) spinlock_profiling.cc:243] Waited 201 ms on lock 0x7b54004506d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:49.238142 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":462354304}
W20260810 18:52:49.271418 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 56331us
W20260810 18:52:49.265384 16748 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43738 (request call id 2217) took 2001 ms (client timeout 1607 ms). Trace:
I20260810 18:52:49.268059 16508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1676) took 1647 ms. Trace:
I20260810 18:52:49.259186 16501 rpcz_store.cc:276] 0810 18:52:47.611127 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:48.536641 (+925514us) service_pool.cc:227] Handling call
0810 18:52:49.208982 (+672341us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.252177 22018 raft_consensus.cc:482] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:48.910925 21810 leader_election.cc:288] T 0c3053ea9475438bab46cba64f4cc2fa P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:49.270993 16506 rpcz_store.cc:276] 0810 18:52:47.611508 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:48.535899 (+924391us) service_pool.cc:227] Handling call
0810 18:52:49.215756 (+679857us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.284960 16506 raft_consensus.cc:1226] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 10 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260810 18:52:49.285359 16631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60066 (request call id 355) took 3136 ms. Trace:
I20260810 18:52:49.285756 16506 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1686) took 1325 ms. Trace:
I20260810 18:52:49.269099 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
W20260810 18:52:49.239976 16742 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43726 (request call id 1644) took 2295 ms (client timeout 1591 ms). Trace:
I20260810 18:52:49.277005 17778 raft_consensus.cc:2379] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 4: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:52:49.268172 16496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1675) took 1660 ms. Trace:
I20260810 18:52:49.264024 21912 raft_consensus.cc:482] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:49.242992 17785 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55586 (request call id 488) took 2273 ms (client timeout 1642 ms). Trace:
I20260810 18:52:49.272612 16743 rpcz_store.cc:276] 0810 18:52:46.960089 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.234087 (+2273998us) service_pool.cc:227] Handling call
0810 18:52:49.234611 (+ 524us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:49.271800 (+ 37189us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.272046 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.279521 16508 rpcz_store.cc:276] 0810 18:52:47.611259 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.226155 (+1614896us) service_pool.cc:227] Handling call
0810 18:52:49.226725 (+ 570us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:49.256971 (+ 30246us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.259020 (+ 2049us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.280507 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
W20260810 18:52:49.278371 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 75965.4us
I20260810 18:52:49.274820 16507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:52330 (request call id 867) took 1430 ms. Trace:
I20260810 18:52:49.282298 21887 raft_consensus.cc:482] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:49.252641 22013 raft_consensus.cc:2781] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Leader election won for term 9
I20260810 18:52:49.285374 21890 leader_election.cc:288] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:49.287425 16495 raft_consensus.cc:1261] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 10 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20260810 18:52:49.290156 17785 rpcz_store.cc:269] 0810 18:52:46.969829 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:46.970000 (+ 171us) service_pool.cc:227] Handling call
0810 18:52:49.242493 (+2272493us) spinlock_profiling.cc:243] Waited 419 ms on lock 0x7b54005401d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:49.242716 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":963271040}
I20260810 18:52:49.289120 16506 rpcz_store.cc:276] 0810 18:52:47.960278 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.283966 (+1323688us) service_pool.cc:227] Handling call
0810 18:52:49.284568 (+ 602us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:49.285424 (+ 856us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.285664 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.309074 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:49.287099 22018 raft_consensus.cc:504] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:49.239459 16623 raft_consensus.cc:2362] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6. Re-sending same reply.
I20260810 18:52:49.287775 21518 leader_election.cc:288] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:49.287523 21948 consensus_queue.cc:237] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:49.288488 16502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1680) took 1552 ms. Trace:
I20260810 18:52:49.285775 21739 raft_consensus.cc:482] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:49.290695 16507 rpcz_store.cc:276] 0810 18:52:47.844396 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.273076 (+1428680us) service_pool.cc:227] Handling call
0810 18:52:49.274724 (+ 1648us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.239879 21444 leader_election.cc:288] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 10 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:49.224639 21170 leader_election.cc:288] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:49.289767 16496 rpcz_store.cc:276] 0810 18:52:47.607316 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.223731 (+1616415us) service_pool.cc:227] Handling call
0810 18:52:49.224204 (+ 473us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:49.267291 (+ 43087us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.268119 (+ 828us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15488}
I20260810 18:52:49.321611 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:49.323348 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:49.323750 21739 raft_consensus.cc:504] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:49.324496 21738 raft_consensus.cc:3032] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:49.324849 16496 raft_consensus.cc:1226] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 6. Current term is 7. Ops: []
I20260810 18:52:49.325074 21738 raft_consensus.cc:729] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:49.295239 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:49.326980 17783 raft_consensus.cc:2454] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 7.
I20260810 18:52:49.286065 16633 raft_consensus.cc:2362] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 8. Re-sending same reply.
I20260810 18:52:49.313238 21644 raft_consensus.cc:3032] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 LEADER]: Stepping down as leader of term 8
I20260810 18:52:49.327594 21926 raft_consensus.cc:3032] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Stepping down as leader of term 7
I20260810 18:52:49.327351 21739 leader_election.cc:288] T 2deb399ca7cb48198e6d3910bef67202 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:49.328619 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:49.300010 16504 raft_consensus.cc:1226] T 694fa0ce1cd246a985705792780adee0 P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 8. Current term is 9. Ops: []
I20260810 18:52:49.300212 21921 raft_consensus.cc:3032] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 8 LEADER]: Stepping down as leader of term 8
I20260810 18:52:49.300769 21901 leader_election.cc:288] T 32567b0896594c1186a56d27e36ca0a0 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:49.301648 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 64576.6us
I20260810 18:52:49.301541 21887 raft_consensus.cc:504] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:49.303421 21912 raft_consensus.cc:504] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:49.338290 21912 leader_election.cc:288] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:49.289582 16742 rpcz_store.cc:269] 0810 18:52:46.945056 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:46.945301 (+ 245us) service_pool.cc:227] Handling call
0810 18:52:49.239524 (+2294223us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b54002ed6d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:49.239783 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":373050240}
W20260810 18:52:49.338632 21889 log.cc:924] Time spent T 8cbe74806d8349e690b392275de70292 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 3.663s user 0.019s sys 0.041s
I20260810 18:52:49.241743 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:49.249046 21808 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:49.306473 21057 raft_consensus.cc:482] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:49.291288 16752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43726 (request call id 1647) took 2331 ms. Trace:
W20260810 18:52:49.279317 16748 rpcz_store.cc:269] 0810 18:52:47.264237 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:47.275765 (+ 11528us) service_pool.cc:227] Handling call
0810 18:52:49.264887 (+1989122us) spinlock_profiling.cc:243] Waited 48.5 ms on lock 0x7b54002c47d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:49.265143 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":111636864}
I20260810 18:52:49.313088 21914 raft_consensus.cc:482] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:49.313803 21952 raft_consensus.cc:482] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:49.314230 21848 leader_election.cc:288] T aa5164e185b84d0a9bf72af61bf78f84 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:49.311713 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 1.68299e+06us
I20260810 18:52:49.308538 16495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52352 (request call id 1734) took 1696 ms. Trace:
I20260810 18:52:49.278442 16745 rpcz_store.cc:276] 0810 18:52:46.069263 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:46.752038 (+682775us) service_pool.cc:227] Handling call
0810 18:52:46.752359 (+ 321us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:47.226067 (+473708us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:47.226103 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:47.226153 (+ 50us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:47.226187 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.241234 (+2015047us) spinlock_profiling.cc:243] Waited 473 ms on lock 0x7b540020b554. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:49.241309 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:49.241661 (+ 352us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1088965376}
I20260810 18:52:49.318339 16502 rpcz_store.cc:276] 0810 18:52:47.735833 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.272012 (+1536179us) service_pool.cc:227] Handling call
0810 18:52:49.272561 (+ 549us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:49.288174 (+ 15613us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.288398 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.327945 21644 raft_consensus.cc:729] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:49.298849 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b33c2c321842498975ee1086991616" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:49.327533 17783 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55602 (request call id 448) took 1695 ms. Trace:
I20260810 18:52:49.393795 17783 rpcz_store.cc:276] 0810 18:52:47.632112 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.294748 (+1662636us) service_pool.cc:227] Handling call
0810 18:52:49.327450 (+ 32702us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.285859 16631 rpcz_store.cc:276] 0810 18:52:46.148538 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:46.151193 (+ 2655us) service_pool.cc:227] Handling call
0810 18:52:46.151636 (+ 443us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:46.151909 (+ 273us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:46.151945 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:46.152502 (+ 557us) log.cc:842] Serialized 26 byte log entry
0810 18:52:46.757994 (+605492us) raft_consensus.cc:1631] Marking committed up to 6
0810 18:52:46.758044 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:46.758113 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:49.221562 (+2463449us) spinlock_profiling.cc:243] Waited 38.3 ms on lock 0x7b44000232cc. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae50e4a95 00007f1ae50e54b2 00007f1ae7d8d511 00007f1ae7d88e1a 00007f1ae7d7cbee 00007f1ae7d7bf36 00007f1ae7d7bb58 00007f1ae7d85551 00007f1ae7d85509 00007f1ae7d85499
0810 18:52:49.284718 (+ 63156us) raft_consensus.cc:1671] finished
0810 18:52:49.284746 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:49.285097 (+ 351us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":88050432,"thread_start_us":604959,"threads_started":1,"wal-append.queue_time_us":594205}
I20260810 18:52:49.315122 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fa0ce1cd246a985705792780adee0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:49.328365 21926 raft_consensus.cc:729] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:49.340050 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:49.331164 21921 raft_consensus.cc:729] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:49.341943 16695 leader_election.cc:302] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:49.352842 21950 raft_consensus.cc:2781] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260810 18:52:49.343951 21883 leader_election.cc:288] T da1fa324d17c479383225a8cea494a80 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:49.346006 17727 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.14.32.195:40171 timed out after 1.660s (SENT) [suppressed 40 similar messages]
I20260810 18:52:49.361728 21705 leader_election.cc:288] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:49.362484 16750 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43726 (request call id 1646) took 2415 ms (client timeout 1824 ms). Trace:
I20260810 18:52:49.357069 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:49.362926 16447 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:49.339659 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
W20260810 18:52:49.411005 16750 rpcz_store.cc:269] 0810 18:52:46.947166 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:47.097964 (+150798us) service_pool.cc:227] Handling call
0810 18:52:49.362077 (+2264113us) spinlock_profiling.cc:243] Waited 27.6 ms on lock 0x7b540009ef54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:49.362280 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":63441280}
I20260810 18:52:49.363834 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:49.412812 16628 raft_consensus.cc:2454] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:49.412902 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:49.413235 21904 log.cc:924] Time spent T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 2.329s user 0.009s sys 0.024s
I20260810 18:52:49.413676 16750 raft_consensus.cc:2454] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 8.
I20260810 18:52:49.413937 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:49.415603 16623 raft_consensus.cc:3037] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:49.416254 21921 consensus_queue.cc:260] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:49.350634 22019 raft_consensus.cc:3032] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 LEADER]: Stepping down as leader of term 8
I20260810 18:52:49.417899 21921 raft_consensus.cc:3037] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:49.417994 22019 raft_consensus.cc:729] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
W20260810 18:52:49.419407 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 77664.4us
I20260810 18:52:49.419525 22019 consensus_queue.cc:260] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:49.421128 22019 raft_consensus.cc:3037] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:49.421552 16695 leader_election.cc:302] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:49.422784 21806 raft_consensus.cc:2781] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:49.422962 21888 consensus_queue.cc:1046] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:52:49.423322 21739 consensus_queue.cc:1046] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:52:49.424270 21888 raft_consensus.cc:2781] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260810 18:52:49.424644 21739 raft_consensus.cc:3032] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:49.344699 16743 raft_consensus.cc:1261] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260810 18:52:49.354362 21738 consensus_queue.cc:260] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:49.425640 21750 raft_consensus.cc:3032] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Stepping down as leader of term 6
I20260810 18:52:49.425981 16743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43726 (request call id 1656) took 2464 ms. Trace:
I20260810 18:52:49.426143 21750 raft_consensus.cc:729] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:49.365758 21901 raft_consensus.cc:482] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:49.366102 17770 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55570 (request call id 573) took 3667 ms. Trace:
I20260810 18:52:49.427604 21901 raft_consensus.cc:504] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:49.315424 22013 raft_consensus.cc:686] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [term 9 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:49.368741 22018 leader_election.cc:288] T 2e86171b8963411f8ae8789b7b1f76f8 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:49.369669 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:49.370954 16747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2202) took 4092 ms. Trace:
I20260810 18:52:49.368757 21038 leader_election.cc:288] T 7dd134517a994065a45633249a3ce405 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
W20260810 18:52:49.306025 16572 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.594s user 0.007s sys 0.012s
I20260810 18:52:49.375922 21871 raft_consensus.cc:3032] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:49.343353 16505 raft_consensus.cc:2454] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 8.
I20260810 18:52:49.293577 16740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43726 (request call id 1653) took 2321 ms. Trace:
I20260810 18:52:49.385664 16752 rpcz_store.cc:276] 0810 18:52:46.950398 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.210381 (+2259983us) service_pool.cc:227] Handling call
0810 18:52:49.210873 (+ 492us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:49.278455 (+ 67582us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.278644 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.392417 16495 rpcz_store.cc:276] 0810 18:52:47.611850 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.226488 (+1614638us) service_pool.cc:227] Handling call
0810 18:52:49.267374 (+ 40886us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:49.307791 (+ 40417us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.308444 (+ 653us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 18:52:49.390312 21853 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.261s user 0.013s sys 0.021s
I20260810 18:52:49.396915 21460 raft_consensus.cc:2726] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:49.397591 17780 raft_consensus.cc:2454] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 8.
I20260810 18:52:49.374485 21549 consensus_queue.cc:260] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:49.429872 16747 rpcz_store.cc:276] 0810 18:52:45.278314 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:45.654243 (+375929us) service_pool.cc:227] Handling call
0810 18:52:46.111476 (+457233us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:46.783159 (+671683us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:47.797034 (+1013875us) spinlock_profiling.cc:243] Waited 44.9 ms on lock 0x7b54002c2ed4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:49.370443 (+1573409us) spinlock_profiling.cc:243] Waited 457 ms on lock 0x7b54002c2ed0. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:49.370734 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1153854464}
I20260810 18:52:49.398399 21808 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:49.399497 21952 raft_consensus.cc:504] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:49.400393 17785 raft_consensus.cc:1226] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260810 18:52:49.400214 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:49.403050 21926 consensus_queue.cc:260] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:49.404946 21704 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:49.404588 21914 raft_consensus.cc:504] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:49.526875 21549 raft_consensus.cc:3037] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:49.527369 21926 raft_consensus.cc:3037] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:49.528641 21914 leader_election.cc:288] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:49.405844 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:49.529354 16745 raft_consensus.cc:3037] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Advancing to term 10
I20260810 18:52:49.530333 21952 leader_election.cc:288] T 8547cf266dd24ff289d7d8ff5e2174d0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:49.428378 17770 rpcz_store.cc:276] 0810 18:52:45.698522 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:45.719952 (+ 21430us) service_pool.cc:227] Handling call
0810 18:52:45.720348 (+ 396us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:45.720698 (+ 350us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:45.720728 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:45.721108 (+ 380us) log.cc:842] Serialized 26 byte log entry
0810 18:52:46.156163 (+435055us) raft_consensus.cc:1631] Marking committed up to 8
0810 18:52:46.930763 (+774600us) log.cc:842] Serialized 14 byte log entry
0810 18:52:47.026433 (+ 95670us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:47.026557 (+ 124us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:49.354010 (+2327453us) spinlock_profiling.cc:243] Waited 3.31 ms on lock 0x7b18004a4e80. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1aee0718b1 00007f1aee07e661 00007f1ae51d2638 00007f1ae51d5b11 00007f1ae51d5ac9 00007f1ae51d5a59 00007f1ae51d5a21 00007f1ae51d4d1d 00007f1aee436204 00007f1aee436039 00007f1ae51b4dc6
0810 18:52:49.365489 (+ 11479us) raft_consensus.cc:1671] finished
0810 18:52:49.365623 (+ 134us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:49.365869 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10328,"prepare.queue_time_us":927215,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":169101,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":39921,"spinlock_wait_cycles":7623424,"thread_start_us":694343,"threads_started":3,"wal-append.queue_time_us":1023119}
I20260810 18:52:49.532308 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02553ba44ccf4839ae0c25e0b3a1b71b" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:49.533447 17770 raft_consensus.cc:3032] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 9 LEADER]: Stepping down as leader of term 9
I20260810 18:52:49.533721 17785 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55586 (request call id 494) took 1761 ms. Trace:
I20260810 18:52:49.533861 17770 raft_consensus.cc:729] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:49.534304 16747 raft_consensus.cc:1226] T 694fa0ce1cd246a985705792780adee0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 8. Current term is 9. Ops: []
I20260810 18:52:49.534219 17785 rpcz_store.cc:276] 0810 18:52:47.765191 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.399480 (+1634289us) service_pool.cc:227] Handling call
0810 18:52:49.399994 (+ 514us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:49.526097 (+126103us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.526338 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.535079 16747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 490) took 2389 ms. Trace:
I20260810 18:52:49.535454 16747 rpcz_store.cc:276] 0810 18:52:47.145238 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.533674 (+2388436us) service_pool.cc:227] Handling call
0810 18:52:49.534034 (+ 360us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:49.534809 (+ 775us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.534989 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.535902 17785 raft_consensus.cc:1261] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260810 18:52:49.400961 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:49.533442 16631 raft_consensus.cc:2396] T 02553ba44ccf4839ae0c25e0b3a1b71b P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260810 18:52:49.537855 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6768c6bbdc345eeb86337a837cc1d22" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:49.538530 17780 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55570 (request call id 581) took 1575 ms. Trace:
I20260810 18:52:49.406698 16633 raft_consensus.cc:3037] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:49.403733 21950 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:49.411880 16742 raft_consensus.cc:3037] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 2 FOLLOWER]: Advancing to term 4
I20260810 18:52:49.424711 21888 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:49.425251 21739 raft_consensus.cc:729] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:49.544217 21950 raft_consensus.cc:3037] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:49.542981 21704 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:49.546118 21739 consensus_queue.cc:260] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:49.426378 16743 rpcz_store.cc:276] 0810 18:52:46.961300 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.329966 (+2368666us) service_pool.cc:227] Handling call
0810 18:52:49.344201 (+ 14235us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:49.425686 (+ 81485us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.425908 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.547586 21704 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:49.426719 21738 raft_consensus.cc:3037] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:49.367029 21886 raft_consensus.cc:482] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:49.555416 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:49.555646 21886 raft_consensus.cc:504] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:49.558171 16745 raft_consensus.cc:2454] T 4e623fc5613147efbb77f01c3b0b7e81 P 980f343dd1fd4e049143f8b9d9f2f1be [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 10.
I20260810 18:52:49.559923 16745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 493) took 2141 ms. Trace:
I20260810 18:52:49.559604 21950 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:49.432220 16628 raft_consensus.cc:2396] T b650a523bfc649d4a7ec8bfd04a67c43 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260810 18:52:49.576088 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:49.577605 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:49.578569 16628 raft_consensus.cc:2354] T bfc23c22a6cb41a5a57a107ebbed91af P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 5. Current term is 6.
I20260810 18:52:49.433362 21871 raft_consensus.cc:729] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:49.580222 21871 consensus_queue.cc:260] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:49.581534 21933 log.cc:924] Time spent T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 2.438s user 0.013s sys 0.025s
I20260810 18:52:49.581904 21871 raft_consensus.cc:3037] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:49.534915 17770 consensus_queue.cc:260] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:49.583498 21739 raft_consensus.cc:3037] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:49.583650 17770 raft_consensus.cc:3037] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Advancing to term 10
W20260810 18:52:49.586309 21963 log.cc:924] Time spent T 7152ca0ee8db485ab76fb9e272bbb59d P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 2.118s user 0.015s sys 0.033s
W20260810 18:52:49.588125 16749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37716 (request call id 497) took 1978 ms (client timeout 1660 ms). Trace:
W20260810 18:52:49.428434 21927 log.cc:924] Time spent T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 2.430s user 0.014s sys 0.019s
I20260810 18:52:49.423295 21806 raft_consensus.cc:482] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:49.427443 21750 consensus_queue.cc:260] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:49.437664 16572 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.800s user 0.008s sys 0.012s
W20260810 18:52:49.438877 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 92946.7us
I20260810 18:52:49.402388 21057 raft_consensus.cc:504] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:49.446972 22018 raft_consensus.cc:2781] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260810 18:52:49.447381 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 84053.1us
W20260810 18:52:49.456385 21839 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.251s user 0.009s sys 0.041s
W20260810 18:52:49.468309 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 96815.1us
W20260810 18:52:49.473976 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 200398us
W20260810 18:52:49.475214 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 78722.3us
I20260810 18:52:49.441916 22013 consensus_queue.cc:237] T 6943a9cdcb8f48dd817a90f357bf54f1 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:49.433805 16740 rpcz_store.cc:276] 0810 18:52:46.960382 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.265288 (+2304906us) service_pool.cc:227] Handling call
0810 18:52:49.265936 (+ 648us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:49.271134 (+ 5198us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.281925 (+ 10791us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.519672 16506 raft_consensus.cc:2396] T b4248aab7d8b4ed0b7c298cad63430f8 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260810 18:52:49.408946 21644 consensus_queue.cc:260] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } attrs { replace: true } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:49.398486 17783 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55586 (request call id 493) took 1642 ms. Trace:
W20260810 18:52:49.522634 16632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 2191) took 2761 ms (client timeout 2210 ms). Trace:
I20260810 18:52:49.523052 16621 raft_consensus.cc:2454] T c3b33c2c321842498975ee1086991616 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:49.525669 16623 raft_consensus.cc:2454] T 7dd134517a994065a45633249a3ce405 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
W20260810 18:52:49.520073 21853 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.391s user 0.015s sys 0.023s
I20260810 18:52:49.538892 17780 rpcz_store.cc:276] 0810 18:52:47.962695 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.537381 (+1574686us) service_pool.cc:227] Handling call
0810 18:52:49.537872 (+ 491us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:49.538193 (+ 321us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:49.538218 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:49.538274 (+ 56us) raft_consensus.cc:1631] Marking committed up to 9
0810 18:52:49.538330 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.538381 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:49.538474 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.358840 21887 leader_election.cc:288] T 9dc14894db5b4872b0a5dd8d933426f4 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:49.560276 16745 rpcz_store.cc:276] 0810 18:52:47.419013 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.558897 (+2139884us) service_pool.cc:227] Handling call
0810 18:52:49.559292 (+ 395us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:49.559593 (+ 301us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:49.559616 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:49.559677 (+ 61us) raft_consensus.cc:1631] Marking committed up to 7
0810 18:52:49.559705 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.559756 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:49.559854 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.536862 21461 leader_election.cc:288] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:49.568276 21886 leader_election.cc:288] T f642e61a7e3c45de883c78d5e5c49712 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:49.569604 21592 raft_consensus.cc:3037] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:52:49.562391 21888 raft_consensus.cc:3037] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:49.570305 16742 raft_consensus.cc:2454] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 4.
I20260810 18:52:49.570510 16636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 2188) took 2802 ms. Trace:
I20260810 18:52:49.571607 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa467f263e640b095b31b6d998258db" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:49.572187 21808 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:49.588642 16749 rpcz_store.cc:269] 0810 18:52:47.609399 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:47.891833 (+282434us) service_pool.cc:227] Handling call
0810 18:52:49.587668 (+1695835us) spinlock_profiling.cc:243] Waited 638 ms on lock 0x7b540034e9d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:49.587877 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1468190464}
I20260810 18:52:49.598237 16631 raft_consensus.cc:2354] T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 8. Current term is 9.
I20260810 18:52:49.605440 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
W20260810 18:52:49.607216 16701 leader_election.cc:334] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 1.990s (SENT)
I20260810 18:52:49.726166 17778 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55602 (request call id 447) took 2097 ms. Trace:
I20260810 18:52:49.740126 16633 raft_consensus.cc:2454] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 5.
I20260810 18:52:49.793062 16623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1760) took 1165 ms. Trace:
I20260810 18:52:49.679250 17781 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55602 (request call id 442) took 2083 ms. Trace:
I20260810 18:52:49.736209 16752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 489) took 2617 ms. Trace:
I20260810 18:52:49.798918 16633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1759) took 1171 ms. Trace:
I20260810 18:52:49.734467 21524 raft_consensus.cc:482] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
I20260810 18:52:49.798877 16623 rpcz_store.cc:276] 0810 18:52:48.628018 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.319683 (+691665us) service_pool.cc:227] Handling call
0810 18:52:49.792980 (+473297us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.729816 16501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52338 (request call id 1683) took 1843 ms. Trace:
I20260810 18:52:49.608290 16500 raft_consensus.cc:2454] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 7.
I20260810 18:52:49.615562 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:49.802559 16626 raft_consensus.cc:3037] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:49.794643 17783 rpcz_store.cc:276] 0810 18:52:47.756045 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.396785 (+1640740us) service_pool.cc:227] Handling call
0810 18:52:49.397470 (+ 685us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:49.397929 (+ 459us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:49.397966 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:49.398054 (+ 88us) raft_consensus.cc:1631] Marking committed up to 8
0810 18:52:49.398123 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.398191 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:49.398382 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.800832 16752 rpcz_store.cc:276] 0810 18:52:47.119025 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.436668 (+2317643us) service_pool.cc:227] Handling call
0810 18:52:49.563125 (+126457us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:49.735692 (+172567us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:49.735735 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:49.735795 (+ 60us) raft_consensus.cc:1631] Marking committed up to 8
0810 18:52:49.735873 (+ 78us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.735954 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:49.736112 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 18:52:49.619961 17787 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55570 (request call id 577) took 2337 ms (client timeout 1857 ms). Trace:
W20260810 18:52:49.796526 16632 rpcz_store.cc:269] 0810 18:52:46.761125 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:46.766176 (+ 5051us) service_pool.cc:227] Handling call
0810 18:52:49.439176 (+2673000us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b5400385ed4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:49.522391 (+ 83215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":327275776}
I20260810 18:52:49.802058 16633 rpcz_store.cc:276] 0810 18:52:48.627722 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.339549 (+711827us) service_pool.cc:227] Handling call
0810 18:52:49.798836 (+459287us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.797138 17778 rpcz_store.cc:276] 0810 18:52:47.628328 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:47.628593 (+ 265us) service_pool.cc:227] Handling call
0810 18:52:49.681974 (+2053381us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b540059e7d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:49.726095 (+ 44121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3190681472}
W20260810 18:52:49.735396 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 185210us
I20260810 18:52:49.616729 16624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 2198) took 2712 ms. Trace:
I20260810 18:52:49.792290 16636 rpcz_store.cc:276] 0810 18:52:46.759997 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:46.760146 (+ 149us) service_pool.cc:227] Handling call
0810 18:52:46.760547 (+ 401us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:46.936843 (+176296us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:46.936886 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:46.936948 (+ 62us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:46.936991 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.464111 (+2527120us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b54001ebc54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:49.517605 (+ 53494us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:49.562540 (+ 44935us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":404760448}
I20260810 18:52:49.801365 17781 rpcz_store.cc:276] 0810 18:52:47.556141 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:47.556336 (+ 195us) service_pool.cc:227] Handling call
0810 18:52:49.620767 (+2064431us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:49.621183 (+ 416us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:49.621226 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:49.621298 (+ 72us) raft_consensus.cc:1631] Marking committed up to 10
0810 18:52:49.621391 (+ 93us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.621476 (+ 85us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:49.622933 (+ 1457us) spinlock_profiling.cc:243] Waited 2.04 s on lock 0x7b54006a1650. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:49.639512 (+ 16579us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4695197952}
W20260810 18:52:49.726752 22030 log.cc:924] Time spent T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea: Append to log took a long time: real 0.114s user 0.001s sys 0.004s
I20260810 18:52:49.602313 21806 raft_consensus.cc:3037] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:49.792904 16701 leader_election.cc:302] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:49.808383 17787 rpcz_store.cc:269] 0810 18:52:47.282559 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:47.282761 (+ 202us) service_pool.cc:227] Handling call
0810 18:52:49.619547 (+2336786us) spinlock_profiling.cc:243] Waited 57.9 ms on lock 0x7b54006a1654. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:49.619743 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":133156096}
W20260810 18:52:49.803565 21839 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.664s user 0.013s sys 0.048s
I20260810 18:52:49.820732 21715 consensus_queue.cc:1035] T cbdf372e5e3b4efb80f86146885fe25e P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
W20260810 18:52:49.820967 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 278830us
I20260810 18:52:49.821100 17771 raft_consensus.cc:2454] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 7.
W20260810 18:52:49.822335 22021 log.cc:924] Time spent T e6768c6bbdc345eeb86337a837cc1d22 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.279s user 0.001s sys 0.000s
W20260810 18:52:49.820771 17775 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55586 (request call id 472) took 6035 ms (client timeout 2201 ms). Trace:
W20260810 18:52:49.844343 17775 rpcz_store.cc:269] 0810 18:52:43.743683 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:43.747331 (+ 3648us) service_pool.cc:227] Handling call
0810 18:52:49.778564 (+6031233us) spinlock_profiling.cc:243] Waited 3.15 s on lock 0x7b540072e554. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:49.778854 (+ 290us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7249748352}
I20260810 18:52:49.823998 21872 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:49.846768 21795 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:49.846702 21685 consensus_queue.cc:1035] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260810 18:52:49.847266 16695 leader_election.cc:302] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:49.824446 17727 leader_election.cc:302] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:49.826030 21986 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:49.799609 21853 raft_consensus.cc:686] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 8 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:49.826635 16508 raft_consensus.cc:1261] T 7ef7c65533724e678b75830002906e2c P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260810 18:52:49.828630 17770 raft_consensus.cc:1261] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 10 FOLLOWER]: Refusing update from remote peer 980f343dd1fd4e049143f8b9d9f2f1be: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260810 18:52:49.829075 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:49.833148 21871 raft_consensus.cc:2726] T 7dd134517a994065a45633249a3ce405 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:49.735375 21901 leader_election.cc:288] T 6a1d591f794a4601bf3c950c2c57b3cf P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
W20260810 18:52:49.618392 21859 log.cc:924] Time spent T b650a523bfc649d4a7ec8bfd04a67c43 P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 4.009s user 0.023s sys 0.038s
I20260810 18:52:49.796900 22018 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:49.725819 21608 raft_consensus.cc:482] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 18:52:49.841969 16701 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 317488us
I20260810 18:52:49.805914 16501 rpcz_store.cc:276] 0810 18:52:47.886693 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.297356 (+1410663us) service_pool.cc:227] Handling call
0810 18:52:49.297967 (+ 611us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:49.298431 (+ 464us) raft_consensus.cc:1520] Early marking committed up to index 4
0810 18:52:49.321440 (+ 23009us) log.cc:842] Serialized 14 byte log entry
0810 18:52:49.728336 (+406896us) spinlock_profiling.cc:243] Waited 1.44 ms on lock 0x7b18002239c0. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1aee0718b1 00007f1aee07e661 00007f1ae51494ca 00007f1aee82af99 00007f1ae500d990 00007f1ae51cf864 00007f1ae51cf155 00007f1aebacf167 00007f1ae8104807 00007f1ae8113f0f 00007f1ae8113e88
0810 18:52:49.728586 (+ 250us) log.cc:842] Serialized 14 byte log entry
0810 18:52:49.729329 (+ 743us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:49.729418 (+ 89us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:49.729503 (+ 85us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.729579 (+ 76us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:49.729724 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":73280,"prepare.queue_time_us":167335,"raft.queue_time_us":35588,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":311,"spinlock_wait_cycles":7847168,"thread_start_us":111477,"threads_started":2,"wal-append.queue_time_us":280438}
I20260810 18:52:49.843209 17785 raft_consensus.cc:3037] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:49.860873 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:49.845921 16628 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 0397d760589d40d3b92ef141cdfc2dbb [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 11.
I20260810 18:52:49.777318 21678 raft_consensus.cc:3032] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Stepping down as leader of term 5
I20260810 18:52:49.867173 21678 raft_consensus.cc:729] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:49.857434 17770 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55586 (request call id 495) took 1378 ms. Trace:
I20260810 18:52:49.867808 17770 rpcz_store.cc:276] 0810 18:52:48.473528 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.532741 (+1059213us) service_pool.cc:227] Handling call
0810 18:52:49.533191 (+ 450us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:49.851096 (+317905us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.851333 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.851902 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:49.609573 21057 leader_election.cc:288] T 42bb15fb54124b8db9b482e3f17fc87c P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:49.853084 16622 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 2202) took 1963 ms (client timeout 1601 ms). Trace:
I20260810 18:52:49.853307 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabdebbbbc4b4fda8760bb69a90a9454" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:49.818921 21644 raft_consensus.cc:3037] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:49.816747 21950 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 9 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:49.803792 21750 raft_consensus.cc:3037] T 6fa467f263e640b095b31b6d998258db P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:49.866092 16628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1778) took 1206 ms. Trace:
I20260810 18:52:49.810434 16624 rpcz_store.cc:276] 0810 18:52:46.904597 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:46.948561 (+ 43964us) service_pool.cc:227] Handling call
0810 18:52:46.964329 (+ 15768us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:46.964799 (+ 470us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:46.964838 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:46.980067 (+ 15229us) log.cc:842] Serialized 26 byte log entry
0810 18:52:47.243311 (+263244us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:47.243349 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:47.243414 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:49.585358 (+2341944us) spinlock_profiling.cc:243] Waited 5.37 ms on lock 0x7b44000232cc. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae50e4a95 00007f1ae50e54b2 00007f1ae7d8d511 00007f1ae7d88e1a 00007f1ae7d7cbee 00007f1ae7d7bf36 00007f1ae7d7bb58 00007f1ae7d85551 00007f1ae7d85509 00007f1ae7d85499
0810 18:52:49.616211 (+ 30853us) raft_consensus.cc:1671] finished
0810 18:52:49.616234 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:49.616551 (+ 317us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":615,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":44097,"spinlock_wait_cycles":12351232,"thread_start_us":213841,"threads_started":1,"wal-append.queue_time_us":435184}
W20260810 18:52:49.875522 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 74007.9us
I20260810 18:52:49.875818 16740 raft_consensus.cc:2454] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 6.
W20260810 18:52:49.876966 16622 rpcz_store.cc:269] 0810 18:52:47.885128 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:47.885306 (+ 178us) service_pool.cc:227] Handling call
0810 18:52:49.819237 (+1933931us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b5400160b54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:49.847950 (+ 28713us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2654287616}
I20260810 18:52:49.891955 16623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1772) took 1236 ms. Trace:
I20260810 18:52:49.861989 22018 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Advancing to term 10
I20260810 18:52:49.894949 16623 rpcz_store.cc:276] 0810 18:52:48.655798 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.818998 (+1163200us) service_pool.cc:227] Handling call
0810 18:52:49.891852 (+ 72854us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.886813 16628 rpcz_store.cc:276] 0810 18:52:49.123503 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.590101 (+466598us) service_pool.cc:227] Handling call
0810 18:52:49.866032 (+275931us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.898298 17727 leader_election.cc:302] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:49.899154 21678 consensus_queue.cc:260] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:49.900863 21678 raft_consensus.cc:3037] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:49.875602 21943 raft_consensus.cc:2781] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:52:49.903035 21926 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:49.903421 21943 raft_consensus.cc:686] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [term 4 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:49.903826 21926 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:49.905055 21943 consensus_queue.cc:237] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:49.906636 21926 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:49.907670 21948 consensus_queue.cc:1035] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260810 18:52:49.871225 21872 raft_consensus.cc:2781] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260810 18:52:49.910207 21872 raft_consensus.cc:482] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:49.910698 21872 raft_consensus.cc:3037] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Advancing to term 10
I20260810 18:52:49.916249 16740 raft_consensus.cc:1226] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 10 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 9. Current term is 10. Ops: []
I20260810 18:52:49.917035 16740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2229) took 2422 ms. Trace:
I20260810 18:52:49.917380 16740 rpcz_store.cc:276] 0810 18:52:47.494474 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.915398 (+2420924us) service_pool.cc:227] Handling call
0810 18:52:49.915903 (+ 505us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:49.916721 (+ 818us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.916969 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.918249 16506 raft_consensus.cc:1226] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 6. Current term is 7. Ops: []
W20260810 18:52:49.920197 16695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 48689us
I20260810 18:52:49.892047 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:49.934175 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:49.926360 16633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1751) took 1301 ms. Trace:
I20260810 18:52:49.936841 16633 rpcz_store.cc:276] 0810 18:52:48.625334 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.836487 (+1211153us) service_pool.cc:227] Handling call
0810 18:52:49.926280 (+ 89793us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.942169 16621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1771) took 1311 ms. Trace:
I20260810 18:52:49.942690 16621 rpcz_store.cc:276] 0810 18:52:48.631033 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.298404 (+667371us) service_pool.cc:227] Handling call
0810 18:52:49.941926 (+643522us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b54003ef3d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:49.942091 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":291725056}
I20260810 18:52:49.935773 16745 raft_consensus.cc:2454] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 7.
I20260810 18:52:49.944703 17727 leader_election.cc:302] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:49.948500 16747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 492) took 2740 ms. Trace:
I20260810 18:52:49.951601 16636 raft_consensus.cc:2354] T cbdf372e5e3b4efb80f86146885fe25e P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 5. Current term is 7.
I20260810 18:52:49.954434 16573 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:52:49.955112 16636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1752) took 1329 ms. Trace:
I20260810 18:52:49.955770 16636 rpcz_store.cc:276] 0810 18:52:48.625673 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.817565 (+1191892us) service_pool.cc:227] Handling call
0810 18:52:49.954952 (+137387us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 18:52:49.958330 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 116650us
I20260810 18:52:49.856810 21524 raft_consensus.cc:504] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:49.964054 16740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 495) took 2456 ms. Trace:
I20260810 18:52:49.964596 16740 rpcz_store.cc:276] 0810 18:52:47.507929 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.918456 (+2410527us) service_pool.cc:227] Handling call
0810 18:52:49.918998 (+ 542us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:49.957540 (+ 38542us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:49.957586 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:49.963695 (+ 6109us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:49.963744 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.963881 (+ 137us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:49.964001 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.960450 21921 raft_consensus.cc:2781] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:49.876173 21608 raft_consensus.cc:504] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:49.847589 21795 raft_consensus.cc:504] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:49.977449 21685 raft_consensus.cc:2781] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:49.963289 21871 raft_consensus.cc:482] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:49.982214 16745 raft_consensus.cc:1261] T 7ef7c65533724e678b75830002906e2c P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260810 18:52:49.977964 21685 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:49.983420 21685 raft_consensus.cc:3037] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:49.969693 21921 raft_consensus.cc:482] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:49.994292 16450 leader_election.cc:302] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters: 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:49.993105 16745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 496) took 2484 ms. Trace:
I20260810 18:52:49.977921 21910 consensus_queue.cc:1035] T d39f8d52fbd04fc58cb3c20d457bab3d P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.021s
I20260810 18:52:50.003708 16745 rpcz_store.cc:276] 0810 18:52:47.508237 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.944317 (+2436080us) service_pool.cc:227] Handling call
0810 18:52:49.979984 (+ 35667us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:49.982783 (+ 2799us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.993036 (+ 10253us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.990777 21871 raft_consensus.cc:504] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:49.997591 21921 raft_consensus.cc:3037] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:50.008997 16740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2230) took 2487 ms. Trace:
I20260810 18:52:50.016211 16740 rpcz_store.cc:276] 0810 18:52:47.522059 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.969993 (+2447934us) service_pool.cc:227] Handling call
0810 18:52:50.000312 (+ 30319us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:50.004739 (+ 4427us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:50.004773 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:50.008702 (+ 3929us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:50.008750 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.008825 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:50.008939 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:50.014724 16633 raft_consensus.cc:1261] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260810 18:52:50.018667 16633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 2213) took 2069 ms. Trace:
I20260810 18:52:50.019047 16633 rpcz_store.cc:276] 0810 18:52:47.948792 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.949517 (+2000725us) service_pool.cc:227] Handling call
0810 18:52:49.982752 (+ 33235us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:50.017469 (+ 34717us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.017672 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:50.019526 16628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 2208) took 2079 ms. Trace:
I20260810 18:52:50.021093 16628 rpcz_store.cc:276] 0810 18:52:47.935681 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.927203 (+1991522us) service_pool.cc:227] Handling call
0810 18:52:49.979087 (+ 51884us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:50.005472 (+ 26385us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:50.005509 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:50.009780 (+ 4271us) raft_consensus.cc:1631] Marking committed up to 9
0810 18:52:50.014886 (+ 5106us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.014992 (+ 106us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:50.015139 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:50.021008 16623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 2206) took 2121 ms. Trace:
I20260810 18:52:50.022228 16623 rpcz_store.cc:276] 0810 18:52:47.900034 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.947052 (+2047018us) service_pool.cc:227] Handling call
0810 18:52:49.998645 (+ 51593us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:50.020614 (+ 21969us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:50.020641 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:50.020736 (+ 95us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:50.020769 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.020845 (+ 76us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:50.020952 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:50.023613 17724 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 980f343dd1fd4e049143f8b9d9f2f1be
W20260810 18:52:50.030572 17724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 73274.4us
I20260810 18:52:50.031875 21549 raft_consensus.cc:2781] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260810 18:52:50.032297 21549 raft_consensus.cc:482] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Starting leader election (detected failure of leader 980f343dd1fd4e049143f8b9d9f2f1be)
I20260810 18:52:50.032670 21549 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Advancing to term 10
I20260810 18:52:50.032879 17724 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:50.033964 21704 consensus_queue.cc:1035] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260810 18:52:50.035398 21901 consensus_queue.cc:1035] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Connected to new peer: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260810 18:52:50.036193 21704 raft_consensus.cc:2781] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260810 18:52:50.036641 21704 raft_consensus.cc:482] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:50.037048 21704 raft_consensus.cc:3037] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:50.040187 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250ae6ba6aa4e55b02199717a2bcde3" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 11 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:50.040244 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e623fc5613147efbb77f01c3b0b7e81" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 9 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:50.042079 16632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1761) took 1413 ms. Trace:
I20260810 18:52:50.043463 16632 rpcz_store.cc:276] 0810 18:52:48.628307 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.819170 (+1190863us) service_pool.cc:227] Handling call
0810 18:52:50.040054 (+220884us) spinlock_profiling.cc:243] Waited 2.06 ms on lock 0x7b5800004298. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae519a851 00007f1ae519abce 000000000034aea9 00007f1aecfc6a30 00007f1aed08c7b4 00007f1aed08c821 00007f1aed03c1b0 00007f1aed03ba94 00007f1ae6723c04 00007f1ae6723b92
0810 18:52:50.041942 (+ 1888us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4731264}
I20260810 18:52:50.023701 21803 consensus_queue.cc:1035] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.021s
I20260810 18:52:50.067942 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:50.045151 16746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2196) took 5206 ms. Trace:
I20260810 18:52:50.051139 21853 consensus_queue.cc:237] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:50.041145 17775 raft_consensus.cc:2396] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260810 18:52:50.057090 21871 leader_election.cc:288] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:50.069319 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:50.070091 16632 raft_consensus.cc:2379] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 7: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:50.072014 16623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 2221) took 2120 ms. Trace:
I20260810 18:52:50.072336 16623 rpcz_store.cc:276] 0810 18:52:47.959653 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:50.023187 (+2063534us) service_pool.cc:227] Handling call
0810 18:52:50.071418 (+ 48231us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:50.071637 (+ 219us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:50.071658 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:50.071695 (+ 37us) raft_consensus.cc:1631] Marking committed up to 8
0810 18:52:50.071764 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.071835 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:50.071940 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:50.073243 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:50.073809 16623 raft_consensus.cc:3037] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:52:50.075331 16622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 2203) took 2189 ms. Trace:
I20260810 18:52:50.075789 16622 rpcz_store.cc:276] 0810 18:52:47.885508 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.933826 (+2048318us) service_pool.cc:227] Handling call
0810 18:52:49.978627 (+ 44801us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:50.000403 (+ 21776us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:50.000435 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:50.000470 (+ 35us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:50.008823 (+ 8353us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.074986 (+ 66163us) spinlock_profiling.cc:243] Waited 16.4 ms on lock 0x7b54003c8054. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:50.075054 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:50.075237 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":37665664}
I20260810 18:52:50.077515 16631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53878 (request call id 1756) took 1450 ms. Trace:
I20260810 18:52:50.077528 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:50.079777 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:50.077901 16631 rpcz_store.cc:276] 0810 18:52:48.626812 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.537515 (+910703us) service_pool.cc:227] Handling call
0810 18:52:50.077338 (+539823us) spinlock_profiling.cc:243] Waited 58.7 ms on lock 0x7b54002a4754. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:50.077457 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":135097728}
I20260810 18:52:50.083786 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:50.084558 16631 raft_consensus.cc:2379] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 9: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:50.085382 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:50.068972 16746 rpcz_store.cc:276] 0810 18:52:44.839026 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:45.185563 (+346537us) service_pool.cc:227] Handling call
0810 18:52:45.185987 (+ 424us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:45.186379 (+ 392us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:45.186408 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:45.186955 (+ 547us) log.cc:842] Serialized 26 byte log entry
0810 18:52:45.323779 (+136824us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:45.323837 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:45.323937 (+ 100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:49.617121 (+4293184us) spinlock_profiling.cc:243] Waited 37.6 ms on lock 0x7b44000232cc. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae5016f11 00007f1ae50e4a95 00007f1ae50e54b2 00007f1ae7d8d511 00007f1ae7d88e1a 00007f1ae7d7cbee 00007f1ae7d7bf36 00007f1ae7d7bb58 00007f1ae7d85551 00007f1ae7d85509 00007f1ae7d85499
0810 18:52:50.044709 (+427588us) raft_consensus.cc:1671] finished
0810 18:52:50.044729 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:50.044964 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":96,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":79,"spinlock_wait_cycles":86435200,"thread_start_us":136368,"threads_started":1,"wal-append.queue_time_us":293127}
I20260810 18:52:50.086719 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:50.087519 16631 raft_consensus.cc:2379] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 9: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:50.088044 16746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2231) took 2563 ms. Trace:
I20260810 18:52:50.088919 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fbe8baa0b345b0b2c4e4489e397660" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:50.066706 17771 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55602 (request call id 445) took 2457 ms (client timeout 1938 ms). Trace:
W20260810 18:52:50.089620 17771 rpcz_store.cc:269] 0810 18:52:47.609092 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:47.609862 (+ 770us) service_pool.cc:227] Handling call
0810 18:52:50.066494 (+2456632us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b540057fdd4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:50.066629 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3833953536}
I20260810 18:52:50.090061 21921 raft_consensus.cc:504] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:50.081318 16345 catalog_manager.cc:5495] T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 6 to 9, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196). New cstate: current_term: 9 leader_uuid: "831925dd8b644ebab7eb0d672a94eddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:52:50.081303 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:50.092679 16622 raft_consensus.cc:1226] T b1af6c49790c42fc9b6455aec5f0d6af P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260810 18:52:50.020988 21839 raft_consensus.cc:2781] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260810 18:52:50.093402 21839 raft_consensus.cc:482] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:50.093441 16622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1765) took 1464 ms. Trace:
I20260810 18:52:50.089758 21795 leader_election.cc:288] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:50.094369 16622 rpcz_store.cc:276] 0810 18:52:48.629415 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:50.091859 (+1462444us) service_pool.cc:227] Handling call
0810 18:52:50.092350 (+ 491us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:50.093163 (+ 813us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.093373 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:49.948943 16747 rpcz_store.cc:276] 0810 18:52:47.208354 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.555093 (+2346739us) service_pool.cc:227] Handling call
0810 18:52:49.555447 (+ 354us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:49.555827 (+ 380us) raft_consensus.cc:1520] Early marking committed up to index 7
0810 18:52:49.599378 (+ 43551us) log.cc:842] Serialized 14 byte log entry
0810 18:52:49.948089 (+348711us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:49.948155 (+ 66us) raft_consensus.cc:1631] Marking committed up to 7
0810 18:52:49.948215 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.948306 (+ 91us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:49.948439 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":12962,"thread_start_us":378393,"threads_started":2}
I20260810 18:52:50.094990 21524 leader_election.cc:288] T 2deb399ca7cb48198e6d3910bef67202 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:50.088454 16746 rpcz_store.cc:276] 0810 18:52:47.524945 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:50.087104 (+2562159us) service_pool.cc:227] Handling call
0810 18:52:50.087493 (+ 389us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:50.087711 (+ 218us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:50.087742 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:50.087783 (+ 41us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:50.087810 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.087865 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:50.087982 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:50.089046 16631 raft_consensus.cc:1226] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260810 18:52:50.096379 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:50.096568 16631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1763) took 1467 ms. Trace:
I20260810 18:52:50.097198 16747 raft_consensus.cc:3037] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:50.097219 16631 rpcz_store.cc:276] 0810 18:52:48.628856 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:50.088323 (+1459467us) service_pool.cc:227] Handling call
0810 18:52:50.088815 (+ 492us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:50.096275 (+ 7460us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.096496 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:50.065343 21549 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:50.063697 21872 raft_consensus.cc:504] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:50.100756 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:50.089680 16740 raft_consensus.cc:2454] T 50fbe8baa0b345b0b2c4e4489e397660 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5.
W20260810 18:52:50.102352 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 126589us
I20260810 18:52:50.099200 16448 leader_election.cc:302] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:50.103868 16621 raft_consensus.cc:1261] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260810 18:52:50.104285 16621 pending_rounds.cc:76] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb: Aborting all ops after (but not including) 3
I20260810 18:52:50.104494 16621 pending_rounds.cc:97] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260810 18:52:50.104748 16621 raft_consensus.cc:2866] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:52:50.094142 21839 raft_consensus.cc:3037] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:50.105299 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:50.107201 17785 raft_consensus.cc:2454] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 7.
I20260810 18:52:50.107237 16745 raft_consensus.cc:2379] T abdb4e808ce3454ab9a52051b369e5d0 P 980f343dd1fd4e049143f8b9d9f2f1be [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 4: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:52:50.070976 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:50.123837 17783 raft_consensus.cc:1226] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 10 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 9. Current term is 10. Ops: []
I20260810 18:52:50.113912 16631 raft_consensus.cc:1226] T 867cf3e179de4be2acd5230d0a3e0ebb P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 5. Current term is 6. Ops: []
I20260810 18:52:50.119920 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:50.110230 16632 raft_consensus.cc:2379] T 03cb267b222b440981c94d431e7273bf P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 7: Already voted for candidate 0397d760589d40d3b92ef141cdfc2dbb in this term.
I20260810 18:52:50.127614 16450 leader_election.cc:302] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:50.118901 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef7c65533724e678b75830002906e2c" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:50.129197 17780 raft_consensus.cc:2379] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be in current term 8: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:52:50.127055 16626 raft_consensus.cc:2454] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 7.
I20260810 18:52:50.108124 21986 raft_consensus.cc:2741] T 03cb267b222b440981c94d431e7273bf P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260810 18:52:50.121304 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32567b0896594c1186a56d27e36ca0a0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:50.131825 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:50.131969 16622 raft_consensus.cc:1226] T 6fa467f263e640b095b31b6d998258db P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 6. Current term is 7. Ops: [4.3-6.4]
I20260810 18:52:50.113139 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb15fb54124b8db9b482e3f17fc87c" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:50.133705 16631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1767) took 1503 ms. Trace:
I20260810 18:52:50.101670 21921 leader_election.cc:288] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:50.136988 16749 raft_consensus.cc:2354] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for earlier term 9. Current term is 10.
W20260810 18:52:50.138936 16626 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53870 (request call id 2209) took 2196 ms (client timeout 2803 ms). Trace:
I20260810 18:52:50.112972 21715 raft_consensus.cc:2726] T ccee0891d8df4f81beb2cfbe1b5485f2 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260810 18:52:50.139446 16626 rpcz_store.cc:269] 0810 18:52:47.942377 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.571097 (+1628720us) service_pool.cc:227] Handling call
0810 18:52:50.132411 (+561314us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:50.136642 16631 rpcz_store.cc:276] 0810 18:52:48.629932 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:50.098205 (+1468273us) service_pool.cc:227] Handling call
0810 18:52:50.113719 (+ 15514us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:50.131885 (+ 18166us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.133648 (+ 1763us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:50.151964 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4248aab7d8b4ed0b7c298cad63430f8" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:50.152119 16633 raft_consensus.cc:1226] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [term 9 LEADER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260810 18:52:50.156414 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:50.160128 16633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1768) took 1529 ms. Trace:
I20260810 18:52:50.160574 16633 rpcz_store.cc:276] 0810 18:52:48.630236 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:50.110299 (+1480063us) service_pool.cc:227] Handling call
0810 18:52:50.145539 (+ 35240us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:50.159784 (+ 14245us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.160059 (+ 275us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 18:52:50.162982 22038 log.cc:924] Time spent T ccee0891d8df4f81beb2cfbe1b5485f2 P 0397d760589d40d3b92ef141cdfc2dbb: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260810 18:52:50.164006 17785 raft_consensus.cc:2454] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 7.
I20260810 18:52:50.170027 16740 raft_consensus.cc:1261] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 11 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260810 18:52:50.170791 16740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2234) took 2611 ms. Trace:
I20260810 18:52:50.173816 16740 rpcz_store.cc:276] 0810 18:52:47.559203 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:50.107389 (+2548186us) service_pool.cc:227] Handling call
0810 18:52:50.146317 (+ 38928us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:50.170495 (+ 24178us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.170730 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 18:52:50.171396 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 166008us
I20260810 18:52:50.133617 21806 raft_consensus.cc:504] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:50.179615 21806 leader_election.cc:288] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:50.180269 16695 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:50.183799 21549 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 10 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:50.187150 21948 raft_consensus.cc:2781] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:52:50.187351 21739 raft_consensus.cc:2781] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:50.187812 21948 raft_consensus.cc:482] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:50.187852 21739 raft_consensus.cc:482] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:50.188369 21948 raft_consensus.cc:3037] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:50.188426 21739 raft_consensus.cc:3037] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:50.192205 16622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1766) took 1562 ms. Trace:
I20260810 18:52:50.192741 16622 rpcz_store.cc:276] 0810 18:52:48.629676 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:50.095436 (+1465760us) service_pool.cc:227] Handling call
0810 18:52:50.131618 (+ 36182us) raft_consensus.cc:1457] Updating replica for 2 ops
0810 18:52:50.134312 (+ 2694us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.192025 (+ 57713us) spinlock_profiling.cc:243] Waited 35.8 ms on lock 0x7b540035dc50. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:50.192151 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":82321280}
I20260810 18:52:50.187088 16746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2233) took 2630 ms. Trace:
I20260810 18:52:50.195072 16746 rpcz_store.cc:276] 0810 18:52:47.556359 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:50.096467 (+2540108us) service_pool.cc:227] Handling call
0810 18:52:50.102921 (+ 6454us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:50.110715 (+ 7794us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:50.110750 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:50.110805 (+ 55us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:50.110868 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.110928 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:50.186870 (+ 75942us) spinlock_profiling.cc:243] Waited 6.12 ms on lock 0x7b540009e550. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f9a09 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:50.186996 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14082816}
I20260810 18:52:50.208721 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:50.208554 16506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c758b1e1139941a9a8c4e703cc058061" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:50.202247 16628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 2220) took 2251 ms. Trace:
I20260810 18:52:50.213804 16632 raft_consensus.cc:1226] T 2e86171b8963411f8ae8789b7b1f76f8 P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Rejecting Update request from peer 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260810 18:52:50.214744 16632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53878 (request call id 1774) took 1558 ms. Trace:
I20260810 18:52:50.215344 16632 rpcz_store.cc:276] 0810 18:52:48.656799 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:50.130034 (+1473235us) service_pool.cc:227] Handling call
0810 18:52:50.167644 (+ 37610us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:50.214429 (+ 46785us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.214681 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:50.219107 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:50.219903 16504 raft_consensus.cc:2454] T 867cf3e179de4be2acd5230d0a3e0ebb P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
I20260810 18:52:50.182965 21704 raft_consensus.cc:504] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:50.220786 16504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:50.221114 16447 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:50.221169 16450 leader_election.cc:302] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:50.221921 21704 leader_election.cc:288] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 9 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:50.223608 22018 raft_consensus.cc:504] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:50.225356 21926 raft_consensus.cc:2781] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260810 18:52:50.225798 21926 raft_consensus.cc:482] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:50.226271 21926 raft_consensus.cc:3037] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:50.228778 22018 leader_election.cc:288] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 10 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:50.229712 22034 log.cc:924] Time spent T cbdf372e5e3b4efb80f86146885fe25e P 980f343dd1fd4e049143f8b9d9f2f1be: Append to log took a long time: real 0.212s user 0.000s sys 0.001s
I20260810 18:52:50.219928 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:50.230978 16495 raft_consensus.cc:2396] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260810 18:52:50.219135 21685 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:50.231843 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:50.232146 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e86171b8963411f8ae8789b7b1f76f8" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:50.232627 16495 raft_consensus.cc:2379] T cdfc652dd163415b94d3f0bc3931a07f P d304ac47fdf846de81337057a4ef7bea [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd in current term 10: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:50.232848 21685 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:50.232997 16749 raft_consensus.cc:2396] T 2e86171b8963411f8ae8789b7b1f76f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260810 18:52:50.233392 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:50.233503 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
I20260810 18:52:50.233934 16495 raft_consensus.cc:3037] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:50.234237 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:50.234999 16448 leader_election.cc:302] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:50.235497 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:50.236127 16749 raft_consensus.cc:2362] T b4082c2e6a774acfa97ec268831748a5 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 5. Re-sending same reply.
I20260810 18:52:50.237035 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152ca0ee8db485ab76fb9e272bbb59d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:50.237100 16448 leader_election.cc:302] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:50.236814 16345 catalog_manager.cc:5495] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } } }
I20260810 18:52:50.237813 16749 raft_consensus.cc:2379] T 7152ca0ee8db485ab76fb9e272bbb59d P 980f343dd1fd4e049143f8b9d9f2f1be [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 10: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:50.239311 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:50.211170 16628 rpcz_store.cc:276] 0810 18:52:47.950869 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:50.022085 (+2071216us) service_pool.cc:227] Handling call
0810 18:52:50.128816 (+106731us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:50.201807 (+ 72991us) raft_consensus.cc:1520] Early marking committed up to index 5
0810 18:52:50.201839 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:50.201887 (+ 48us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:50.201940 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.202019 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:50.202176 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":15118,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":7181}
I20260810 18:52:50.240615 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:50.241412 16746 raft_consensus.cc:2379] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 5: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:50.231679 16621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 2218) took 2281 ms. Trace:
I20260810 18:52:50.242970 16750 raft_consensus.cc:1261] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260810 18:52:50.243571 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681439c754d942b7847d73744abbe269" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea" is_pre_election: true
I20260810 18:52:50.238690 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1d591f794a4601bf3c950c2c57b3cf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:50.243764 17771 raft_consensus.cc:1226] T ccee0891d8df4f81beb2cfbe1b5485f2 P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 8. Current term is 9. Ops: []
I20260810 18:52:50.244310 16743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2220) took 2978 ms. Trace:
I20260810 18:52:50.243376 16621 rpcz_store.cc:276] 0810 18:52:47.950264 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.951010 (+2000746us) service_pool.cc:227] Handling call
0810 18:52:49.981904 (+ 30894us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:50.105148 (+123244us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.231434 (+126286us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b54001ec154. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:50.231599 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":275650304}
I20260810 18:52:50.219132 16500 raft_consensus.cc:1261] T ef3f6c02785c4a76ba768b20f41f2dcd P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Refusing update from remote peer 0397d760589d40d3b92ef141cdfc2dbb: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260810 18:52:50.243196 17727 leader_election.cc:398] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 9 election: Vote denied by peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171) with higher term. Message: Invalid argument: T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for earlier term 9. Current term is 10.
I20260810 18:52:50.249109 17727 leader_election.cc:401] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260810 18:52:50.231631 16508 raft_consensus.cc:1226] T e6768c6bbdc345eeb86337a837cc1d22 P d304ac47fdf846de81337057a4ef7bea [term 9 LEADER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 8. Current term is 9. Ops: []
I20260810 18:52:50.245134 16623 raft_consensus.cc:2454] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 4.
I20260810 18:52:50.249531 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95820d00c5148f2bc4169d3aafe28dd" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:50.250432 16450 leader_election.cc:302] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea; no voters:
I20260810 18:52:50.242658 16746 raft_consensus.cc:1261] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Refusing update from remote peer d304ac47fdf846de81337057a4ef7bea: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260810 18:52:50.239341 16345 catalog_manager.cc:5495] T 32567b0896594c1186a56d27e36ca0a0 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 6 to 7, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196). New cstate: current_term: 7 leader_uuid: "831925dd8b644ebab7eb0d672a94eddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:52:50.258690 16746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2237) took 2412 ms. Trace:
I20260810 18:52:50.259245 16746 rpcz_store.cc:276] 0810 18:52:47.846589 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:50.241951 (+2395362us) service_pool.cc:227] Handling call
0810 18:52:50.242301 (+ 350us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:50.254406 (+ 12105us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.258614 (+ 4208us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:50.244491 21739 raft_consensus.cc:504] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:50.262272 16505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d304ac47fdf846de81337057a4ef7bea"
W20260810 18:52:50.261428 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 51445.6us
I20260810 18:52:50.262214 16504 raft_consensus.cc:2454] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 7.
I20260810 18:52:50.265211 16745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2236) took 2410 ms. Trace:
I20260810 18:52:50.268190 16448 leader_election.cc:302] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:50.268011 16745 rpcz_store.cc:276] 0810 18:52:47.846165 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:50.128334 (+2282169us) service_pool.cc:227] Handling call
0810 18:52:50.159508 (+ 31174us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:50.246588 (+ 87080us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:50.246615 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:50.256163 (+ 9548us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:50.256224 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.256304 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:50.256443 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:50.239655 22018 raft_consensus.cc:2726] T 8cbe74806d8349e690b392275de70292 P d304ac47fdf846de81337057a4ef7bea [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:52:50.243937 21962 raft_consensus.cc:2726] T f642e61a7e3c45de883c78d5e5c49712 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260810 18:52:50.273154 16506 raft_consensus.cc:1226] T f250ae6ba6aa4e55b02199717a2bcde3 P d304ac47fdf846de81337057a4ef7bea [term 10 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260810 18:52:50.274351 16750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43726 (request call id 1657) took 3259 ms. Trace:
I20260810 18:52:50.275952 16750 rpcz_store.cc:276] 0810 18:52:47.017306 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.464188 (+2446882us) service_pool.cc:227] Handling call
0810 18:52:49.589213 (+125025us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:50.245995 (+656782us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.274095 (+ 28100us) spinlock_profiling.cc:243] Waited 653 ms on lock 0x7b5400140d54. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:50.274300 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1502807296}
I20260810 18:52:50.247689 16743 rpcz_store.cc:276] 0810 18:52:47.265708 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.547412 (+2281704us) service_pool.cc:227] Handling call
0810 18:52:49.547822 (+ 410us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:49.548335 (+ 513us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:49.548362 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0810 18:52:49.548868 (+ 506us) log.cc:842] Serialized 26 byte log entry
0810 18:52:49.600367 (+ 51499us) raft_consensus.cc:1631] Marking committed up to 2
0810 18:52:49.600603 (+ 236us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:49.600671 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 18:52:50.244086 (+643415us) raft_consensus.cc:1671] finished
0810 18:52:50.244106 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:50.244234 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":19305,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":143,"thread_start_us":51065,"threads_started":1,"wal-append.queue_time_us":420299}
I20260810 18:52:50.281216 16740 raft_consensus.cc:2379] T f95820d00c5148f2bc4169d3aafe28dd P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 5: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:50.263108 21608 leader_election.cc:288] T 6943a9cdcb8f48dd817a90f357bf54f1 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:50.269834 21986 raft_consensus.cc:3037] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Advancing to term 10
W20260810 18:52:50.322638 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 51683.4us
I20260810 18:52:50.202344 21806 consensus_queue.cc:1046] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.065s
I20260810 18:52:50.326761 16750 raft_consensus.cc:1226] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 7. Current term is 8. Ops: [7.8-7.8]
W20260810 18:52:50.327420 16343 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37026 (request call id 227) took 21649 ms (client timeout 15000 ms). Trace:
I20260810 18:52:50.328318 16743 raft_consensus.cc:1226] T f250ae6ba6aa4e55b02199717a2bcde3 P 980f343dd1fd4e049143f8b9d9f2f1be [term 10 LEADER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 9. Current term is 10. Ops: []
I20260810 18:52:50.351966 16743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 500) took 2030 ms. Trace:
I20260810 18:52:50.352397 16743 rpcz_store.cc:276] 0810 18:52:48.322020 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:50.283500 (+1961480us) service_pool.cc:227] Handling call
0810 18:52:50.328135 (+ 44635us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:50.351665 (+ 23530us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.351901 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:50.329427 16746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2238) took 2482 ms. Trace:
I20260810 18:52:50.320385 22018 raft_consensus.cc:2781] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 10 FOLLOWER]: Leader election won for term 10
I20260810 18:52:50.330824 21678 consensus_queue.cc:1035] T f642e61a7e3c45de883c78d5e5c49712 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:52:50.331609 16749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2239) took 2484 ms. Trace:
I20260810 18:52:50.356724 16749 rpcz_store.cc:276] 0810 18:52:47.847316 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:50.264212 (+2416896us) service_pool.cc:227] Handling call
0810 18:52:50.282436 (+ 18224us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:50.331285 (+ 48849us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:50.331314 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:50.331374 (+ 60us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:50.331410 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.331458 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:50.331561 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:50.331832 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7477728b4962a7069618cdf12b92" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:50.333287 16631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:50.334411 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:50.336674 16345 catalog_manager.cc:5495] T c3b33c2c321842498975ee1086991616 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 3 to 5, leader changed from 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195) to 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196). New cstate: current_term: 5 leader_uuid: "831925dd8b644ebab7eb0d672a94eddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:52:50.358201 16633 raft_consensus.cc:2454] T 156c7477728b4962a7069618cdf12b92 P 0397d760589d40d3b92ef141cdfc2dbb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 9.
I20260810 18:52:50.338841 17726 leader_election.cc:302] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:50.340083 21592 consensus_meta.cc:287] T f95820d00c5148f2bc4169d3aafe28dd P d304ac47fdf846de81337057a4ef7bea: Time spent flushing consensus metadata: real 0.533s user 0.011s sys 0.000s
I20260810 18:52:50.342952 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16638c4ef6148f8a1e6ff4e81ae955d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:50.344228 16495 raft_consensus.cc:2396] T 9e960488563344af83b979c260c2dd06 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260810 18:52:50.344637 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33bce85658374382881d7f8d510b407d" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:50.364152 16623 raft_consensus.cc:2454] T d16638c4ef6148f8a1e6ff4e81ae955d P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 6.
I20260810 18:52:50.340112 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbe74806d8349e690b392275de70292" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:50.365150 16745 raft_consensus.cc:2379] T 8cbe74806d8349e690b392275de70292 P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 5: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:50.365453 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4082c2e6a774acfa97ec268831748a5" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:50.354625 16746 rpcz_store.cc:276] 0810 18:52:47.846937 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:50.267792 (+2420855us) service_pool.cc:227] Handling call
0810 18:52:50.328897 (+ 61105us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:50.329125 (+ 228us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:50.329147 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:50.329202 (+ 55us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:50.329236 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.329285 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:50.329375 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:50.356346 16750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2240) took 2508 ms. Trace:
I20260810 18:52:50.367084 16750 rpcz_store.cc:276] 0810 18:52:47.847741 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:50.283556 (+2435815us) service_pool.cc:227] Handling call
0810 18:52:50.326499 (+ 42943us) raft_consensus.cc:1457] Updating replica for 1 ops
0810 18:52:50.356066 (+ 29567us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.356291 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 18:52:50.370216 16345 catalog_manager.cc:5495] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 5 to 6, leader changed from d304ac47fdf846de81337057a4ef7bea (127.14.32.193) to 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196). New cstate: current_term: 6 leader_uuid: "831925dd8b644ebab7eb0d672a94eddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:52:50.372040 16621 raft_consensus.cc:2454] T 33bce85658374382881d7f8d510b407d P 0397d760589d40d3b92ef141cdfc2dbb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 8.
I20260810 18:52:50.263752 21888 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:50.408701 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:50.402195 16740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2243) took 1035 ms. Trace:
I20260810 18:52:50.375123 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b6af1458cd44d8ba26b43579613ecb" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:50.372107 16631 raft_consensus.cc:2454] T 0a481385212f4e189ea5b2f5e215be4d P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 7.
I20260810 18:52:50.381080 16633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee0891d8df4f81beb2cfbe1b5485f2" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb"
I20260810 18:52:50.376899 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:50.383858 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb6043f3f9f4b38b58e608973ac019c" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:50.384204 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac7251004484853a9326a55ffbb9c6d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:50.412258 16749 raft_consensus.cc:2362] T dac7251004484853a9326a55ffbb9c6d P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 8. Re-sending same reply.
I20260810 18:52:50.382519 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1fa324d17c479383225a8cea494a80" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:50.387100 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ec62e59fcf42ddaf65f9e6b21b49e6" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
W20260810 18:52:50.350878 16343 rpcz_store.cc:269] 0810 18:52:28.678323 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:28.682330 (+ 4007us) service_pool.cc:227] Handling call
0810 18:52:50.327151 (+21644821us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 18:52:45.281198 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 18:52:45.360650 (+ 79452us) write_op.cc:432] Acquiring schema lock in shared mode
0810 18:52:45.360704 (+ 54us) write_op.cc:435] Acquired schema lock
0810 18:52:45.360729 (+ 25us) tablet.cc:660] Decoding operations
0810 18:52:45.361588 (+ 859us) write_op.cc:620] Acquiring the partition lock for write op
0810 18:52:45.361672 (+ 84us) write_op.cc:641] Partition lock acquired for write op
0810 18:52:45.361709 (+ 37us) tablet.cc:683] Acquiring locks for 4 operations
0810 18:52:45.362206 (+ 497us) tablet.cc:699] Row locks acquired
0810 18:52:45.362254 (+ 48us) write_op.cc:260] PREPARE: finished
0810 18:52:45.362454 (+ 200us) write_op.cc:270] Start()
0810 18:52:45.362598 (+ 144us) write_op.cc:276] Timestamp: P: 1786387965362391 usec, L: 0
0810 18:52:45.362626 (+ 28us) op_driver.cc:331] REPLICATION: starting
0810 18:52:45.481235 (+118609us) log.cc:842] Serialized 1696 byte log entry
0810 18:52:47.509674 (+2028439us) spinlock_profiling.cc:243] Waited 3.73 ms on lock 0x7b1800286060. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1aee0718b1 00007f1aee07e661 00007f1ae7d8982c 00007f1ae51b49dc 00007f1aedfeba39 00007f1ae51d18bb 00007f1ae51cfcbf 00007f1ae51d1e22 00007f1ae7d7b5bb 00007f1ae7d81063 00007f1ae7d81411
0810 18:52:47.510494 (+ 820us) op_driver.cc:444] REPLICATION: finished
0810 18:52:47.849934 (+339440us) write_op.cc:301] APPLY: starting
0810 18:52:47.850047 (+ 113us) tablet.cc:1365] starting BulkCheckPresence
0810 18:52:47.850238 (+ 191us) tablet.cc:1368] finished BulkCheckPresence
0810 18:52:47.850261 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 18:52:47.850935 (+ 674us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 18:52:47.895776 (+ 44841us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0810 18:52:47.935668 (+ 39892us) write_op.cc:312] APPLY: finished
0810 18:52:47.943556 (+ 7888us) log.cc:842] Serialized 41 byte log entry
0810 18:52:47.943740 (+ 184us) write_op.cc:489] Releasing partition, row and schema locks
0810 18:52:47.943850 (+ 110us) write_op.cc:454] Released schema lock
0810 18:52:47.943891 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0810 18:52:47.944254 (+ 363us) write_op.cc:489] Releasing partition, row and schema locks
0810 18:52:47.944281 (+ 27us) write_op.cc:454] Released schema lock
0810 18:52:49.298480 (+1354199us) spinlock_profiling.cc:243] Waited 480 ms on lock 0x7b54000028d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80f7689 00007f1ae8095804 00007f1ae80957b3 00007f1ae8095728 00007f1ae80956e3 00007f1ae80949cc 00007f1ae80bea16 00007f1ae80a65cb 00007f1ae8090c8d 00007f1ae8094665
Metrics: {"reported_tablets":49,"child_traces":[["op",{"apply.queue_time_us":339107,"apply.run_cpu_time_us":2790,"apply.run_wall_time_us":94638,"num_ops":4,"prepare.queue_time_us":680514,"prepare.run_cpu_time_us":66458,"prepare.run_wall_time_us":2228860,"raft.queue_time_us":445813,"raft.run_cpu_time_us":86512,"raft.run_wall_time_us":2271087,"replication_time_us":2147744,"spinlock_wait_cycles":1113010944,"thread_start_us":2544079,"threads_started":4,"wal-append.queue_time_us":909262,"wal-append.run_cpu_time_us":113781,"wal-append.run_wall_time_us":2557675}]]}
I20260810 18:52:50.373049 21549 raft_consensus.cc:2726] T c758b1e1139941a9a8c4e703cc058061 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260810 18:52:50.363415 21872 leader_election.cc:288] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 10 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:50.402835 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3efea4006f43ee933f50d3e328a1f0" candidate_uuid: "831925dd8b644ebab7eb0d672a94eddd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:50.386096 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6b0b676405daf5ffddc50a4d21f" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:50.420106 16750 raft_consensus.cc:2396] T 8810b6b0b676405daf5ffddc50a4d21f P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260810 18:52:50.415746 16743 raft_consensus.cc:2379] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 10: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:50.410087 16740 rpcz_store.cc:276] 0810 18:52:49.366942 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:50.348405 (+981463us) service_pool.cc:227] Handling call
0810 18:52:50.373183 (+ 24778us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:50.401765 (+ 28582us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 18:52:50.401810 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:50.401871 (+ 61us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:50.401935 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.402021 (+ 86us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:50.402140 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2583,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":15387}
I20260810 18:52:50.377887 22018 raft_consensus.cc:686] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [term 10 LEADER]: Becoming Leader. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
I20260810 18:52:50.424880 22018 consensus_queue.cc:237] T 4e623fc5613147efbb77f01c3b0b7e81 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 7.4, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:50.431022 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a481385212f4e189ea5b2f5e215be4d" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:50.414002 16747 raft_consensus.cc:2454] T 9e960488563344af83b979c260c2dd06 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 8.
I20260810 18:52:50.435575 16748 raft_consensus.cc:1226] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 8. Current term is 9. Ops: []
W20260810 18:52:50.436905 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 98150.1us
I20260810 18:52:50.438802 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af6c49790c42fc9b6455aec5f0d6af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:50.440114 16740 raft_consensus.cc:2396] T b1af6c49790c42fc9b6455aec5f0d6af P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260810 18:52:50.435711 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b650a523bfc649d4a7ec8bfd04a67c43" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:50.444484 16750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3053ea9475438bab46cba64f4cc2fa" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:50.447582 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a93f1f1e1cc4e038b5ab9280d80cb23" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:50.449487 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:50.450543 16746 raft_consensus.cc:2379] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 8: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:50.433858 21715 raft_consensus.cc:2781] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260810 18:52:50.433797 21592 raft_consensus.cc:2726] T b1af6c49790c42fc9b6455aec5f0d6af P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260810 18:52:50.454397 16745 raft_consensus.cc:2454] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P 980f343dd1fd4e049143f8b9d9f2f1be [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 11.
I20260810 18:52:50.453061 21739 leader_election.cc:288] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:50.407387 21926 raft_consensus.cc:504] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:50.462224 17727 leader_election.cc:302] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:50.453723 16626 raft_consensus.cc:2454] T 3f3efea4006f43ee933f50d3e328a1f0 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831925dd8b644ebab7eb0d672a94eddd in term 7.
I20260810 18:52:50.476125 16749 raft_consensus.cc:1226] T fabdebbbbc4b4fda8760bb69a90a9454 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 7. Current term is 8. Ops: []
I20260810 18:52:50.475318 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:50.480234 17783 raft_consensus.cc:3037] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:50.482874 16743 raft_consensus.cc:1226] T ef3f6c02785c4a76ba768b20f41f2dcd P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 7. Current term is 8. Ops: []
I20260810 18:52:50.473940 16345 catalog_manager.cc:5495] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 5 to 6, leader changed from d304ac47fdf846de81337057a4ef7bea (127.14.32.193) to 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196). New cstate: current_term: 6 leader_uuid: "831925dd8b644ebab7eb0d672a94eddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:52:50.492053 16748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37716 (request call id 491) took 3303 ms. Trace:
I20260810 18:52:50.492519 16748 rpcz_store.cc:276] 0810 18:52:47.188627 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.537717 (+2349090us) service_pool.cc:227] Handling call
0810 18:52:49.538091 (+ 374us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:50.435983 (+897892us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.491766 (+ 55783us) spinlock_profiling.cc:243] Waited 897 ms on lock 0x7b54002ee5d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80faf51 00007f1ae80f999e 00007f1aed03b2e9 00007f1ae6721684 00007f1ae6721612 00007f1ae6721541 00007f1ae67214bc 00007f1ae6720772 00007f1ae652494c 00007f1ae6523d86
0810 18:52:50.491987 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2063580032}
W20260810 18:52:50.496210 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 50710.9us
I20260810 18:52:50.498769 16628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8547cf266dd24ff289d7d8ff5e2174d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:50.501107 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb4e808ce3454ab9a52051b369e5d0" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0397d760589d40d3b92ef141cdfc2dbb" is_pre_election: true
I20260810 18:52:50.502048 16622 raft_consensus.cc:2354] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980f343dd1fd4e049143f8b9d9f2f1be for earlier term 3. Current term is 4.
I20260810 18:52:50.444379 21962 consensus_queue.cc:1046] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.043s
I20260810 18:52:50.484956 21704 raft_consensus.cc:2781] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader election won for term 8
I20260810 18:52:50.445729 21948 raft_consensus.cc:504] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:50.506817 16624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 2211) took 2552 ms. Trace:
I20260810 18:52:50.512317 17726 leader_election.cc:302] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:50.515334 21853 consensus_queue.cc:1046] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20260810 18:52:50.520116 16502 raft_consensus.cc:2379] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 6: Already voted for candidate d304ac47fdf846de81337057a4ef7bea in this term.
I20260810 18:52:50.510736 21715 raft_consensus.cc:482] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:50.537487 21715 raft_consensus.cc:3037] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:50.548549 21986 raft_consensus.cc:2726] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171) with higher term. Message: Invalid argument: T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 831925dd8b644ebab7eb0d672a94eddd for earlier term 9. Current term is 10.
I20260810 18:52:50.526166 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:50.527978 21608 raft_consensus.cc:3032] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 LEADER]: Stepping down as leader of term 9
I20260810 18:52:50.509125 16624 rpcz_store.cc:276] 0810 18:52:47.948123 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.925484 (+1977361us) service_pool.cc:227] Handling call
0810 18:52:49.968598 (+ 43114us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:50.010031 (+ 41433us) raft_consensus.cc:1520] Early marking committed up to index 3
0810 18:52:50.046349 (+ 36318us) log.cc:842] Serialized 14 byte log entry
0810 18:52:50.500407 (+454058us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:50.500474 (+ 67us) raft_consensus.cc:1631] Marking committed up to 3
0810 18:52:50.500534 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.500626 (+ 92us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:50.500765 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":27519,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":45,"raft.queue_time_us":9448,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":3919,"thread_start_us":444180,"threads_started":1}
I20260810 18:52:50.535847 21926 leader_election.cc:288] T d39f8d52fbd04fc58cb3c20d457bab3d P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:50.536255 21962 raft_consensus.cc:3032] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 9 LEADER]: Stepping down as leader of term 9
W20260810 18:52:50.548804 17786 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55586 (request call id 497) took 2074 ms (client timeout 1550 ms). Trace:
I20260810 18:52:50.548861 17770 raft_consensus.cc:1226] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [term 8 LEADER]: Rejecting Update request from peer d304ac47fdf846de81337057a4ef7bea for earlier term 7. Current term is 8. Ops: []
I20260810 18:52:50.518793 16345 catalog_manager.cc:5495] T 694fa0ce1cd246a985705792780adee0 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 3 to 8. New cstate: current_term: 8 leader_uuid: "831925dd8b644ebab7eb0d672a94eddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:52:50.513896 21704 raft_consensus.cc:686] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [term 8 LEADER]: Becoming Leader. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
W20260810 18:52:50.557837 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 490551us
W20260810 18:52:50.554805 17786 rpcz_store.cc:269] 0810 18:52:48.474274 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:48.610536 (+136262us) service_pool.cc:227] Handling call
0810 18:52:50.548414 (+1937878us) spinlock_profiling.cc:243] Waited 1.89 s on lock 0x7b54000348d4. stack: 00007f1ae51a0033 00007f1ae519ff30 00007f1ae4afe534 00007f1aee80a8b4 00007f1aee80a849 00007f1ae807f471 00007f1ae80fe2ae 00007f1aed03bc27 00007f1ae6723c04 00007f1ae6723b92 00007f1ae6723ac1 00007f1ae6723a3c 00007f1ae6722cf2 00007f1ae652494c 00007f1ae6523d86 00007f1ae652372f
0810 18:52:50.548634 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4343296384}
I20260810 18:52:50.509337 21839 raft_consensus.cc:504] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:50.525113 21888 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 9 election: Requested vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
W20260810 18:52:50.565964 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 297875us
I20260810 18:52:50.562336 16345 catalog_manager.cc:5495] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd reported cstate change: term changed from 5 to 6, leader changed from d304ac47fdf846de81337057a4ef7bea (127.14.32.193) to 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196). New cstate: current_term: 6 leader_uuid: "831925dd8b644ebab7eb0d672a94eddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 18:52:50.564640 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf372e5e3b4efb80f86146885fe25e" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:50.564837 16636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53870 (request call id 2219) took 2614 ms. Trace:
I20260810 18:52:50.554082 21608 raft_consensus.cc:729] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:50.553260 21549 raft_consensus.cc:2726] T 8810b6b0b676405daf5ffddc50a4d21f P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260810 18:52:50.556890 21962 raft_consensus.cc:729] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d304ac47fdf846de81337057a4ef7bea, State: Running, Role: LEADER
W20260810 18:52:50.577214 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 67559.9us
I20260810 18:52:50.580267 16500 raft_consensus.cc:1226] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Rejecting Update request from peer 831925dd8b644ebab7eb0d672a94eddd for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260810 18:52:50.569473 16636 rpcz_store.cc:276] 0810 18:52:47.950558 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.971159 (+2020601us) service_pool.cc:227] Handling call
0810 18:52:50.020020 (+ 48861us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:50.130812 (+110792us) raft_consensus.cc:1520] Early marking committed up to index 8
0810 18:52:50.153345 (+ 22533us) log.cc:842] Serialized 14 byte log entry
0810 18:52:50.564431 (+411086us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:50.564496 (+ 65us) raft_consensus.cc:1631] Marking committed up to 8
0810 18:52:50.564556 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.564638 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:50.564771 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":17003,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":40,"raft.queue_time_us":4913,"thread_start_us":364398,"threads_started":1}
I20260810 18:52:50.586695 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc14894db5b4872b0a5dd8d933426f4" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:50.585604 21704 consensus_queue.cc:237] T 9e960488563344af83b979c260c2dd06 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:50.603294 17771 raft_consensus.cc:2454] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 831925dd8b644ebab7eb0d672a94eddd [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 10.
I20260810 18:52:50.588878 21795 raft_consensus.cc:3032] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 8 LEADER]: Stepping down as leader of term 8
I20260810 18:52:50.609385 17726 leader_election.cc:302] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:50.584051 21608 consensus_queue.cc:260] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:50.612341 16695 leader_election.cc:302] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:50.615913 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc23c22a6cb41a5a57a107ebbed91af" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:50.616621 16743 raft_consensus.cc:3037] T bfc23c22a6cb41a5a57a107ebbed91af P 980f343dd1fd4e049143f8b9d9f2f1be [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:52:50.619129 21608 raft_consensus.cc:3037] T cdfc652dd163415b94d3f0bc3931a07f P 980f343dd1fd4e049143f8b9d9f2f1be [term 9 FOLLOWER]: Advancing to term 10
I20260810 18:52:50.621143 21948 leader_election.cc:288] T bfc23c22a6cb41a5a57a107ebbed91af P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 6 election: Requested vote from peers 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:50.624051 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867cf3e179de4be2acd5230d0a3e0ebb" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:50.624871 16747 raft_consensus.cc:2454] T 867cf3e179de4be2acd5230d0a3e0ebb P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 6.
I20260810 18:52:50.618979 21962 consensus_queue.cc:260] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:50.634781 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cb267b222b440981c94d431e7273bf" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:50.635612 16749 raft_consensus.cc:3037] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Advancing to term 7
I20260810 18:52:50.613382 21839 leader_election.cc:288] T 649d43610ba24875a4418e39b25126ba P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 9 election: Requested vote from peers 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171), 831925dd8b644ebab7eb0d672a94eddd (127.14.32.196:33387)
I20260810 18:52:50.613806 21795 raft_consensus.cc:729] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 831925dd8b644ebab7eb0d672a94eddd, State: Running, Role: LEADER
I20260810 18:52:50.619851 21888 raft_consensus.cc:2781] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 10 FOLLOWER]: Leader election won for term 10
W20260810 18:52:50.640583 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 280360us
I20260810 18:52:50.636696 21549 raft_consensus.cc:2726] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260810 18:52:50.650628 21678 consensus_queue.cc:1046] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260810 18:52:50.651094 22051 log.cc:924] Time spent T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.056s user 0.000s sys 0.001s
W20260810 18:52:50.654899 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 69426.6us
I20260810 18:52:50.645574 21921 raft_consensus.cc:482] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d304ac47fdf846de81337057a4ef7bea)
W20260810 18:52:50.658346 17726 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 49034.8us
I20260810 18:52:50.659602 17726 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd; no voters:
I20260810 18:52:50.657192 21921 raft_consensus.cc:504] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:50.660849 16747 raft_consensus.cc:1226] T d16638c4ef6148f8a1e6ff4e81ae955d P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Rejecting Update request from peer 0397d760589d40d3b92ef141cdfc2dbb for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260810 18:52:50.648927 21888 raft_consensus.cc:686] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [term 10 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
W20260810 18:52:50.671118 16447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.196:33387, user_credentials={real_user=slave}} blocked reactor thread for 448496us
I20260810 18:52:50.670980 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb399ca7cb48198e6d3910bef67202" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:50.675030 16575 leader_election.cc:334] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Timed out: RequestConsensusVote RPC to 127.14.32.195:40171 timed out after 1.541s (SENT)
I20260810 18:52:50.675647 16575 leader_election.cc:302] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:50.670773 16573 leader_election.cc:302] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0397d760589d40d3b92ef141cdfc2dbb; no voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:50.684209 16624 raft_consensus.cc:1261] T abdb4e808ce3454ab9a52051b369e5d0 P 0397d760589d40d3b92ef141cdfc2dbb [term 4 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260810 18:52:50.683202 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ad39135ed48d5adab628ba706791a" candidate_uuid: "0397d760589d40d3b92ef141cdfc2dbb" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:50.687458 16747 raft_consensus.cc:2379] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0397d760589d40d3b92ef141cdfc2dbb in current term 8: Already voted for candidate 980f343dd1fd4e049143f8b9d9f2f1be in this term.
I20260810 18:52:50.692380 21685 raft_consensus.cc:3032] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 LEADER]: Stepping down as leader of term 7
I20260810 18:52:50.692911 21685 raft_consensus.cc:729] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0397d760589d40d3b92ef141cdfc2dbb, State: Running, Role: LEADER
I20260810 18:52:50.694016 21685 consensus_queue.cc:260] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:50.672381 21549 raft_consensus.cc:2781] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 18:52:50.711402 16742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2228) took 3232 ms. Trace:
I20260810 18:52:50.715142 16749 raft_consensus.cc:2454] T 03cb267b222b440981c94d431e7273bf P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0397d760589d40d3b92ef141cdfc2dbb in term 7.
I20260810 18:52:50.712602 16742 rpcz_store.cc:276] 0810 18:52:47.479457 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.819024 (+2339567us) service_pool.cc:227] Handling call
0810 18:52:49.861766 (+ 42742us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:49.952068 (+ 90302us) raft_consensus.cc:1520] Early marking committed up to index 4
0810 18:52:50.441555 (+489487us) log.cc:842] Serialized 14 byte log entry
0810 18:52:50.711014 (+269459us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:50.711082 (+ 68us) raft_consensus.cc:1631] Marking committed up to 4
0810 18:52:50.711126 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.711212 (+ 86us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:50.711342 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15760,"prepare.queue_time_us":100540,"prepare.run_cpu_time_us":22,"prepare.run_wall_time_us":23,"raft.queue_time_us":10927,"thread_start_us":641519,"threads_started":2}
I20260810 18:52:50.718019 16447 leader_election.cc:302] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb
I20260810 18:52:50.703934 21678 raft_consensus.cc:2726] T f95820d00c5148f2bc4169d3aafe28dd P 0397d760589d40d3b92ef141cdfc2dbb [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:52:50.712548 21549 raft_consensus.cc:482] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:52:50.711874 21962 raft_consensus.cc:3037] T 7152ca0ee8db485ab76fb9e272bbb59d P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Advancing to term 10
I20260810 18:52:50.683147 21948 consensus_queue.cc:1035] T 7a93f1f1e1cc4e038b5ab9280d80cb23 P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.045s
I20260810 18:52:50.743492 21678 raft_consensus.cc:2726] T 4c87160448924498ac7de68947b3dc19 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260810 18:52:50.744264 21678 raft_consensus.cc:482] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:52:50.744907 16448 leader_election.cc:302] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 980f343dd1fd4e049143f8b9d9f2f1be
I20260810 18:52:50.735232 21549 raft_consensus.cc:3037] T 3f3efea4006f43ee933f50d3e328a1f0 P 831925dd8b644ebab7eb0d672a94eddd [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:50.752259 16742 raft_consensus.cc:1261] T b4248aab7d8b4ed0b7c298cad63430f8 P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Refusing update from remote peer 831925dd8b644ebab7eb0d672a94eddd: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260810 18:52:50.762703 21704 consensus_queue.cc:1035] T b4248aab7d8b4ed0b7c298cad63430f8 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260810 18:52:50.763222 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 93149.4us
W20260810 18:52:50.765583 16575 leader_election.cc:334] T 8810b6b0b676405daf5ffddc50a4d21f P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171): Timed out: RequestConsensusVote RPC to 127.14.32.195:40171 timed out after 1.633s (SENT)
I20260810 18:52:50.741843 21685 raft_consensus.cc:3037] T fabdebbbbc4b4fda8760bb69a90a9454 P 0397d760589d40d3b92ef141cdfc2dbb [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:50.729455 21715 raft_consensus.cc:504] T dac7251004484853a9326a55ffbb9c6d P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
I20260810 18:52:50.716310 21795 consensus_queue.cc:260] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:50.783222 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e960488563344af83b979c260c2dd06" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd" is_pre_election: true
I20260810 18:52:50.744763 21678 raft_consensus.cc:504] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 } } peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } }
W20260810 18:52:50.791366 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 63457.1us
I20260810 18:52:50.793124 16448 leader_election.cc:302] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d304ac47fdf846de81337057a4ef7bea; no voters: 0397d760589d40d3b92ef141cdfc2dbb, 831925dd8b644ebab7eb0d672a94eddd
I20260810 18:52:50.796830 17783 raft_consensus.cc:2454] T 192ad39135ed48d5adab628ba706791a P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980f343dd1fd4e049143f8b9d9f2f1be in term 8.
I20260810 18:52:50.798517 21678 leader_election.cc:288] T 1cb6043f3f9f4b38b58e608973ac019c P 0397d760589d40d3b92ef141cdfc2dbb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 980f343dd1fd4e049143f8b9d9f2f1be (127.14.32.195:40171)
I20260810 18:52:50.801580 16695 leader_election.cc:302] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd, 980f343dd1fd4e049143f8b9d9f2f1be; no voters:
I20260810 18:52:50.802940 21872 raft_consensus.cc:2781] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 FOLLOWER]: Leader election won for term 8
I20260810 18:52:50.803445 21872 raft_consensus.cc:686] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [term 8 LEADER]: Becoming Leader. State: Replica: 980f343dd1fd4e049143f8b9d9f2f1be, State: Running, Role: LEADER
I20260810 18:52:50.804318 21872 consensus_queue.cc:237] T 192ad39135ed48d5adab628ba706791a P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
W20260810 18:52:50.808061 16450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 44994.9us
I20260810 18:52:50.816725 17783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f8d52fbd04fc58cb3c20d457bab3d" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:50.821030 17783 raft_consensus.cc:3037] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Advancing to term 9
I20260810 18:52:50.815932 16752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43738 (request call id 2227) took 3342 ms. Trace:
W20260810 18:52:50.827747 16701 connection.cc:232] RPC call timeout handler was delayed by -1.324570873985067s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19930000000000003s for a response
W20260810 18:52:50.831215 16701 connection.cc:232] RPC call timeout handler was delayed by -1.2513291030190885s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2392s for a response
W20260810 18:52:50.832237 16701 leader_election.cc:334] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827): Timed out: RequestConsensusVote RPC to 127.14.32.194:45827 timed out after 2.056s (SENT)
I20260810 18:52:50.832687 16701 leader_election.cc:302] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980f343dd1fd4e049143f8b9d9f2f1be; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
W20260810 18:52:50.818430 22059 log.cc:924] Time spent T f250ae6ba6aa4e55b02199717a2bcde3 P 831925dd8b644ebab7eb0d672a94eddd: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260810 18:52:50.836256 17724 leader_election.cc:302] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 980f343dd1fd4e049143f8b9d9f2f1be, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:50.839046 17724 leader_election.cc:302] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 831925dd8b644ebab7eb0d672a94eddd; no voters: 0397d760589d40d3b92ef141cdfc2dbb, d304ac47fdf846de81337057a4ef7bea
I20260810 18:52:50.839437 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc652dd163415b94d3f0bc3931a07f" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:50.811380 22064 raft_consensus.cc:2781] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260810 18:52:50.843261 17771 raft_consensus.cc:2379] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d304ac47fdf846de81337057a4ef7bea in current term 10: Already voted for candidate 831925dd8b644ebab7eb0d672a94eddd in this term.
I20260810 18:52:50.844784 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649d43610ba24875a4418e39b25126ba" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "831925dd8b644ebab7eb0d672a94eddd"
I20260810 18:52:50.811399 22053 raft_consensus.cc:2726] T fabdebbbbc4b4fda8760bb69a90a9454 P d304ac47fdf846de81337057a4ef7bea [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260810 18:52:50.823084 21921 leader_election.cc:288] T 4c87160448924498ac7de68947b3dc19 P 831925dd8b644ebab7eb0d672a94eddd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d304ac47fdf846de81337057a4ef7bea (127.14.32.193:40791), 0397d760589d40d3b92ef141cdfc2dbb (127.14.32.194:45827)
I20260810 18:52:50.752104 21888 consensus_queue.cc:237] T 04ec62e59fcf42ddaf65f9e6b21b49e6 P 980f343dd1fd4e049143f8b9d9f2f1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 } } peers { permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 } } peers { permanent_uuid: "831925dd8b644ebab7eb0d672a94eddd" member_type: VOTER last_known_addr { host: "127.14.32.196" port: 33387 } attrs { promote: false } }
I20260810 18:52:50.769134 21943 consensus_queue.cc:1035] T abdb4e808ce3454ab9a52051b369e5d0 P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0397d760589d40d3b92ef141cdfc2dbb" member_type: VOTER last_known_addr { host: "127.14.32.194" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
I20260810 18:52:50.838553 21057 raft_consensus.cc:2726] T 3f3efea4006f43ee933f50d3e328a1f0 P d304ac47fdf846de81337057a4ef7bea [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260810 18:52:50.837517 21795 raft_consensus.cc:3037] T 649d43610ba24875a4418e39b25126ba P 831925dd8b644ebab7eb0d672a94eddd [term 8 FOLLOWER]: Advancing to term 9
W20260810 18:52:50.862278 16573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.193:40791, user_credentials={real_user=slave}} blocked reactor thread for 193781us
I20260810 18:52:50.811236 22062 consensus_queue.cc:1035] T d16638c4ef6148f8a1e6ff4e81ae955d P d304ac47fdf846de81337057a4ef7bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" member_type: VOTER last_known_addr { host: "127.14.32.195" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.123s
I20260810 18:52:50.840818 21806 raft_consensus.cc:2726] T 02553ba44ccf4839ae0c25e0b3a1b71b P 980f343dd1fd4e049143f8b9d9f2f1be [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260810 18:52:50.868525 16575 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.195:40171, user_credentials={real_user=slave}} blocked reactor thread for 102960us
I20260810 18:52:50.846515 21704 raft_consensus.cc:2726] T 681439c754d942b7847d73744abbe269 P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260810 18:52:50.846871 22064 raft_consensus.cc:482] T ab9362a1df854c4fad0204ee2f9f78ad P d304ac47fdf846de81337057a4ef7bea [term 9 FOLLOWER]: Starting leader election (detected failure of leader 0397d760589d40d3b92ef141cdfc2dbb)
I20260810 18:52:50.833870 21853 consensus_queue.cc:1035] T 7ef7c65533724e678b75830002906e2c P 831925dd8b644ebab7eb0d672a94eddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d304ac47fdf846de81337057a4ef7bea" member_type: VOTER last_known_addr { host: "127.14.32.193" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.021s
I20260810 18:52:50.828043 16752 rpcz_store.cc:276] 0810 18:52:47.473626 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:52:49.814158 (+2340532us) service_pool.cc:227] Handling call
0810 18:52:49.869230 (+ 55072us) raft_consensus.cc:1457] Updating replica for 0 ops
0810 18:52:49.928933 (+ 59703us) raft_consensus.cc:1520] Early marking committed up to index 5
0810 18:52:50.452979 (+524046us) log.cc:842] Serialized 14 byte log entry
0810 18:52:50.812089 (+359110us) log.cc:842] Serialized 14 byte log entry
0810 18:52:50.815608 (+ 3519us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0810 18:52:50.815659 (+ 51us) raft_consensus.cc:1631] Marking committed up to 5
0810 18:52:50.815703 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 18:52:50.815768 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 18:52:50.815876 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11657,"prepare.queue_time_us":133408,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":83,"raft.queue_time_us":9907,"raft.run_cpu_time_us":345,"raft.run_wall_time_us":26553,"thread_start_us":798965,"threads_started":3,"wal-append.queue_time_us":337577}
W20260810 18:52:50.880326 16448 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.32.194:45827, user_credentials={real_user=slave}} blocked reactor thread for 87299.7us
I20260810 18:52:50.866617 17783 raft_consensus.cc:2454] T d39f8d52fbd04fc58cb3c20d457bab3d P 831925dd8b644ebab7eb0d672a94eddd [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d304ac47fdf846de81337057a4ef7bea in term 9.
I20260810 18:52:50.850966 21986 raft_consensus.cc:2726] T cdfc652dd163415b94d3f0bc3931a07f P 831925dd8b644ebab7eb0d672a94eddd [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260810 18:52:50.891999 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5164e185b84d0a9bf72af61bf78f84" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be"
I20260810 18:52:50.892673 16749 raft_consensus.cc:3037] T aa5164e185b84d0a9bf72af61bf78f84 P 980f343dd1fd4e049143f8b9d9f2f1be [term 7 FOLLOWER]: Advancing to term 8
I20260810 18:52:50.899585 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3f6c02785c4a76ba768b20f41f2dcd" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
I20260810 18:52:50.901858 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd134517a994065a45633249a3ce405" candidate_uuid: "d304ac47fdf846de81337057a4ef7bea" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "980f343dd1fd4e049143f8b9d9f2f1be" is_pre_election: true
W20260810 18:52:50.902994 17727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus ->